Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ChIRAAG: ChatGPT Informed Rapid and Automated Assertion Generation (2402.00093v3)

Published 31 Jan 2024 in cs.SE and cs.LG

Abstract: System Verilog Assertion (SVA) formulation -- a critical yet complex task is a prerequisite in the Assertion Based Verification (ABV) process. Traditionally, SVA formulation involves expert-driven interpretation of specifications, which is time-consuming and prone to human error. Recently, LLM-informed automatic assertion generation is gaining interest. We designed a novel framework called ChIRAAG, based on OpenAI GPT4, to generate SVA from natural language specifications of a design. ChIRAAG constitutes the systematic breakdown of design specifications into a standardized format, further generating assertions from formatted specifications using LLM. Furthermore, we used few test cases to validate the LLM-generated assertions. Automatic feedback of log messages from the simulation tool to the LLM ensures that the framework can generate correct SVAs. In our experiments, only 27% of LLM-generated raw assertions had errors, which was rectified in few iterations based on the simulation log. Our results on OpenTitan designs show that LLMs can streamline and assist engineers in the assertion generation process, reshaping verification workflows.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. P. Sharma et al., “Performance of chatgpt on usmle: Unlocking the potential of large language models for ai-assisted medical education,” arXiv preprint: 2307.00112, 2023.
  2. S. Elkins et al., “How useful are educational questions generated by large language models?” in International Conference on Artificial Intelligence in Education.   Springer, 2023.
  3. S. Wu et al., “Bloomberggpt: A large language model for finance,” arXiv preprint arXiv:2303.17564, 2023.
  4. X. Zhang et al., “Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters,” in Proceedings ACM Conf on Information and Knowledge Management, 2023.
  5. H. Foster, “Part 1: The 2020 wilson research group functional verification study,” Feb 2021. [Online]. Available: https://tinyurl.com/verification-foster-group
  6. H. Witharana et al., “Automated generation of security assertions for rtl models,” ACM Journal on Emerging Technologies in Computing Systems, vol. 19, no. 1, pp. 1–27, 2023.
  7. A. F. V. T. Committee et al., “Psl property specification language reference manual,” 2004.
  8. D. Wang et al., “Automatic assume guarantee analysis for assertion-based formal verification,” in Proc. Asia and South Pacific Design Automation Conference, 2005, pp. 561–566.
  9. R. Rosenfeld, “Two decades of statistical language modeling: Where do we go from here?” Proceedings of the IEEE, 2000.
  10. J. Gao et al., “Introduction to the special issue on statistical language modeling,” 2004.
  11. W. X. Zhao et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023.
  12. A. J. Thirunavukarasu et al., “Large language models in medicine,” Nature medicine, 2023.
  13. S. Gilbert et al., “Large language model ai chatbots require approval as medical devices,” Nature Medicine, 2023.
  14. E. Waisberg et al., “Large language model (llm)-driven chatbots for neuro-ophthalmic medical education,” Eye, 2023.
  15. K. M. Jablonka et al., “14 examples of how llms can transform materials science and chemistry: a reflection on a large language model hackathon,” Digital Discovery, 2023.
  16. A. N. Rubungo et al., “Llm-prop: Predicting physical and electronic properties of crystalline solids from their text descriptions,” arXiv preprint arXiv:2310.14029, 2023.
  17. C. Wang et al., “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,” arXiv preprint arXiv:2310.07521, 2023.
  18. H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
  19. OpenAI, “Gpt-4 technical report,” 2023.
  20. M. Xu, “Medicalgpt: Training medical gpt model,” https://github.com/shibing624/MedicalGPT, 2023.
  21. Z. Qi et al., “Foodgpt: A large language model in food testing domain with incremental pre-training and knowledge graph prompt,” 2023.
  22. Z. Zhang et al., “Balancing specialized and general skills in llms: The impact of modern tuning and data strategy,” 2023.
  23. B. Roziere et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023.
  24. R. Li et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161, 2023.
  25. M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  26. Y. Li, “Competition-level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022.
  27. Y. Deng et al., “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,” arXiv preprint arXiv:2304.02014, 2023.
  28. C. S. Xia et al., “Universal fuzzing via large language models,” arXiv preprint arXiv:2308.04748, 2023.
  29. C. S. Xia, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proceedings of ACM Joint European Software Engineering Conference and Symposium on Foundations of Software Engineering, 2022.
  30. N. Jiang et al., “Impact of code language models on automated program repair,” arXiv preprint arXiv:2302.05020, 2023.
  31. M. Orenes-Vera, “Using llms to facilitate formal verification of rtl,” 2023.
  32. R. Kande et al., “Llm-assisted generation of hardware assertions,” 2023.
  33. OpenTitan, “OpenTitan Repository,” https://opentitan.org/, Visited: 2023.
  34. B. Mali, “LLM-Generated-SVA,” https://github.com/BhabeshMali/LLM-Generated-SVA, 2023.
  35. P. Srikumar, “Fast and wrong: The case for formally specifying hardware with llms,” in Proc. ACM ASPLOS, 2023.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now