Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMs (2402.00386v3)

Published 1 Feb 2024 in cs.AR

Abstract: Assertion-based verification (ABV) is a critical method for ensuring design circuits comply with their architectural specifications, which are typically described in natural language. This process often requires human interpretation by verification engineers to convert these specifications into functional verification assertions. Existing methods for generating assertions from natural language specifications are limited to sentences extracted by engineers, discouraging its practical application. In this work, we present AssertLLM, an automatic assertion generation framework that processes complete specification files. AssertLLM breaks down the complex task into three phases, incorporating three customized LLMs for extracting structural specifications, mapping signal definitions, and generating assertions. Our evaluation of AssertLLM on a full design, encompassing 23 I/O signals, demonstrates that 89\% of the generated assertions are both syntactically and functionally accurate.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. H. Witharana, Y. Lyu, S. Charles, and P. Mishra, “A survey on assertion-based hardware verification,” ACM Computing Surveys (CSUR), vol. 54, no. 11s, pp. 1–33, 2022.
  2. S. Germiniani and G. Pravadelli, “Harm: a hint-based assertion miner,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 41, no. 11, pp. 4277–4288, 2022.
  3. A. Danese, N. D. Riva, and G. Pravadelli, “A-team: Automatic template-based assertion miner,” in Proceedings of the 54th Annual Design Automation Conference 2017, 2017, pp. 1–6.
  4. S. Vasudevan, D. Sheridan, S. Patel, D. Tcheng, B. Tuohy, and D. Johnson, “Goldmine: Automatic assertion generation using data mining and static analysis,” in 2010 Design, Automation & Test in Europe Conference & Exhibition (DATE 2010).   IEEE, 2010, pp. 626–629.
  5. M. Orenes-Vera, A. Manocha, D. Wentzlaff, and M. Martonosi, “Autosva: Democratizing formal verification of rtl module interactions,” in 2021 58th ACM/IEEE Design Automation Conference (DAC).   IEEE, 2021, pp. 535–540.
  6. W. Fang, G. Hu, and H. Zhang, “r-map: Relating implementation and specification in hardware refinement checking,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023.
  7. R. Kande, H. Pearce, B. Tan, B. Dolan-Gavitt, S. Thakur, R. Karri, and J. Rajendran, “Llm-assisted generation of hardware assertions,” arXiv preprint arXiv:2306.14027, 2023.
  8. M. Orenes-Vera, M. Martonosi, and D. Wentzlaff, “Using llms to facilitate formal verification of rtl,” arXiv e-prints, pp. arXiv–2309, 2023.
  9. C. Sun, C. Hahn, and C. Trippel, “Towards improving verification productivity with circuit-aware translation of natural language to systemverilog assertions,” in First International Workshop on Deep Learning-aided Verification (DAV), 2023.
  10. C. B. Harris and I. G. Harris, “Glast: Learning formal grammars to translate natural language specifications into hardware assertions,” in 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE).   IEEE, 2016, pp. 966–971.
  11. R. Krishnamurthy and M. S. Hsiao, “Controlled natural language framework for generating assertions from hardware specifications,” in 2019 IEEE 13th International Conference on Semantic Computing (ICSC).   IEEE, 2019, pp. 367–370.
  12. J. Zhao and I. G. Harris, “Automatic assertion generation from natural language specifications using subtree analysis,” in 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).   IEEE, 2019, pp. 598–601.
  13. R. Krishnamurthy and M. S. Hsiao, “Ease: Enabling hardware assertion synthesis from english,” in Rules and Reasoning: Third International Joint Conference, RuleML+ RR 2019, Bolzano, Italy, September 16–19, 2019, Proceedings 3.   Springer, 2019, pp. 82–96.
  14. S. J. Frederiksen, J. Aromando, and M. S. Hsiao, “Automated assertion generation from natural language specifications,” in 2020 IEEE International Test Conference (ITC).   IEEE, 2020, pp. 1–5.
  15. O. Keszocze and I. G. Harris, “Chatbot-based assertion generation from natural language specifications,” in 2019 Forum for Specification and Design Languages (FDL).   IEEE, 2019, pp. 1–6.
  16. G. Parthasarathy, S. Nanda, P. Choudhary, and P. Patil, “Spectosva: Circuit specification document to systemverilog assertion translation,” in 2021 Second Document Intelligence Workshop at KDD.
  17. F. Aditi and M. S. Hsiao, “Hybrid rule-based and machine learning system for assertion generation from natural language specifications,” in 2022 IEEE 31st Asian Test Symposium (ATS).   IEEE, 2022, pp. 126–131.
  18. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  19. S. Liu, W. Fang, Y. Lu, Q. Zhang, H. Zhang, and Z. Xie, “Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,” arXiv preprint arXiv:2312.08617, 2023.
  20. J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Challenges and opportunities in conversational hardware design,” arXiv preprint arXiv:2305.13243, 2023.
  21. Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” arXiv preprint arXiv:2308.05345, 2023.
  22. M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” arXiv preprint arXiv:2309.07544, 2023.
  23. S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan-Gavitt, and S. Garg, “Benchmarking large language models for automated verilog rtl code generation,” in DATE, 2023.
  24. S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “Autochip: Automating hdl generation using llm feedback,” arXiv preprint arXiv:2311.04887, 2023.
  25. M. Nair, R. Sadhukhan, and D. Mukhopadhyay, “Generating secure hardware using chatgpt resistant to cwes,” Cryptology ePrint Archive, 2023.
  26. M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al., “Chipnemo: Domain-adapted llms for chip design,” arXiv preprint arXiv:2311.00176, 2023.
  27. Y. Tsai, M. Liu, and H. Ren, “Rtlfixer: Automatically fixing rtl syntax errors with large language models,” arXiv preprint arXiv:2311.16543, 2023.
  28. Z. He, H. Wu, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” in MLCAD Workshop, 2023.
  29. Y. Fu, Y. Zhang, Z. Yu, S. Li, Z. Ye, C. Li, C. Wan, and Y. Lin, “Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models,” arXiv preprint arXiv:2309.10730, 2023.
  30. Z. Yan, Y. Qin, X. S. Hu, and Y. Shi, “On the viability of using llms for sw/hw co-design: An example in designing cim dnn accelerators,” arXiv preprint arXiv:2306.06923, 2023.
  31. B. Ahmad, S. Thakur, B. Tan, R. Karri, and H. Pearce, “Fixing hardware security bugs with large language models,” arXiv preprint arXiv:2302.01215, 2023.
  32. M. Li, W. Fang, Q. Zhang, and Z. Xie, “Specllm: Exploring generation and review of vlsi design specification with large language model,” arXiv preprint arXiv:2401.13266, 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Wenji Fang (13 papers)
  2. Mengming Li (7 papers)
  3. Min Li (246 papers)
  4. Zhiyuan Yan (81 papers)
  5. Shang Liu (68 papers)
  6. Hongce Zhang (10 papers)
  7. Zhiyao Xie (30 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now