Large Language Models to Generate System-Level Test Programs Targeting Non-functional Properties (2403.10086v2)
Abstract: System-Level Test (SLT) has been a part of the test flow for integrated circuits for over a decade and still gains importance. However, no systematic approaches exist for test program generation, especially targeting non-functional properties of the Device under Test (DUT). Currently, test engineers manually compose test suites from off-the-shelf software, approximating the end-user environment of the DUT. This is a challenging and tedious task that does not guarantee sufficient control over non-functional properties. This paper proposes LLMs to generate test programs. We take a first glance at how pre-trained LLMs perform in test program generation to optimize non-functional properties of the DUT. Therefore, we write a prompt to generate C code snippets that maximize the instructions per cycle of a super-scalar, out-of-order architecture in simulation. Additionally, we apply prompt and hyperparameter optimization to achieve the best possible results without further training.
- Harry H. Chen “Beyond structural test, the rising need for system-level test” In International Symposium on VLSI Design, Automation and Test (VLSI-DAT) IEEE, 2018 DOI: 10.1109/VLSI-DAT.2018.8373238
- “Exploring the Mysteries of System-Level Test” In 2020 IEEE 29th Asian Test Symposium (ATS) IEEE, 2020 DOI: 10.1109/ATS49688.2020.9301557
- “GeST: An Automatic Framework For Generating CPU Stress-Tests” In IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2019, Madison, WI, USA, March 24-26, 2019 IEEE, 2019 DOI: 10.1109/ISPASS.2019.00009
- “SonicBOOM: The 3rd Generation Berkeley Out-of-Order Machine” In Fourth Workshop on Computer Architecture Research with RISC-V, 2020
- “Code llama: Open foundation models for code” In arXiv preprint arXiv:2308.12950, 2023
- Nikolaos I. Deligiannis, Riccardo Cantoro and Matteo Sonza Reorda “Automating the Generation of Programs Maximizing the Sustained Switching Activity in Microprocessor units via Evolutionary Techniques” In Microprocessors and Microsystems 98, 2023 DOI: https://doi.org/10.1016/j.micpro.2023.104775
- “Effective SAT-based Solutions for Generating Functional Sequences Maximizing the Sustained Switching Activity in a Pipelined Processor” In 2021 IEEE 30th Asian Test Symposium (ATS), 2021 DOI: 10.1109/ATS52891.2021.00025
- “A Flexible Framework for the Automatic Generation of SBST Programs” In IEEE Transactions on Very Large Scale Integration (VLSI) Systems 24.10 Institute of ElectricalElectronics Engineers (IEEE), 2016 DOI: 10.1109/TVLSI.2016.2538800
- “Automating Greybox System-Level Test Generation” In 2023 IEEE European Test Symposium (ETS), 2023 DOI: 10.1109/ETS56758.2023.10173985
- “Large language models as optimizers” In arXiv preprint arXiv:2309.03409, 2023
- “Large language models are human-level prompt engineers” In arXiv preprint arXiv:2211.01910, 2022
- “Software Testing with Large Language Model: Survey, Landscape, and Vision”, 2023 arXiv:2307.07221 [cs.SE]
- Vaibhav Kumar “vaibkumr/prompt-optimizer”, 2024 URL: https://github.com/vaibkumr/prompt-optimizer
- “Optuna: A Next-generation Hyperparameter Optimization Framework” In CoRR abs/1907.10902, 2019 arXiv:1907.10902
- “Evaluating large language models trained on code” In arXiv preprint arXiv:2107.03374, 2021