Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation (2408.11053v2)

Abstract: The application of large-LLMs to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.

• The paper introduces an enhanced VerilogEval benchmark that integrates specification-to-RTL tasks and a refined error classification system to improve model evaluation.
• The paper applies diverse LLMs, including GPT-4 Turbo and Llama 3.1, demonstrating significant performance boosts in Verilog code generation through in-context learning.
• The paper’s methodology and detailed failure analysis provide actionable insights for optimizing LLM architectures for digital hardware design and RTL synthesis.

Evaluation of Large-LLMs for Verilog Code Generation

The paper "Revisiting VerilogEval: Newer LLMs, In-Context Learning, and Specification-to-RTL Tasks" explores the utility and performance of large-LLMs applied to the task of Verilog hardware description code generation. This task is notably distinct from typical software code generation due to the limited amount of hardware-specific data and the complexity of hardware description languages (HDLs) such as Verilog.

VerilogEval Benchmark Enhancements

The authors of this paper have significantly enhanced the existing VerilogEval benchmark, which was originally created to evaluate LLMs on Verilog code completion tasks. Significant augmentations included support for specification-to-RTL translation tasks, a more nuanced failure classification system for model errors, and infrastructure improvements to facilitate batch processing and in-context learning (ICL) prompts.

• Specification-to-RTL Tasks: The benchmark now evaluates models on tasks that require converting specifications directly into RTL, reflecting a more realistic design environment.
• Failure Classification: A novel and comprehensive error classification system provides insight into common LLM shortcomings, which is a crucial step for understanding model limitations and guiding future model improvements.
• Infrastructure Improvements: By adopting a Makefile-based system, the paper presents a versatile and scalable approach to evaluation, allowing for full visibility into each model's performance through detailed logs and results.

Evaluation of Diverse LLMs

The evaluations compared a wide array of models, including GPT-4 Turbo, Llama 3.1 405B, and RTL-Coder, among others. The results revealed substantial improvements in pass rates over previous models:

• GPT-4 Turbo achieved up to a 59% pass rate in specification-to-RTL tasks, highlighting its superior adaptability to hardware design tasks. It also improved significantly when utilizing ICL techniques.
• Llama 3.1 405B, an open-source model, not only approached but matched the performance of closed-source models, marking a significant milestone in the efficacy of openly accessible LLMs for hardware tasks.
• RTL-Coder, a smaller model specialized for RTL, demonstrated impressive pass rates in comparison to more general-purpose models, reinforcing the benefits of domain-specific training.

Influence of In-Context Learning

In-context learning significantly affected performance, as elucidated in the paper. Notably, the impact varied considerably between models:

• While GPT-4 Turbo showed robust consistency across different ICL settings, other models like Llama 3.1 70B presented mixed results, suggesting potential tuning opportunities tailored to individual model architectures.
• The variability underscores the necessity of continued research into better prompt engineering and context-specific adaptations to maximize the efficacy of ICL.

Implications and Future Directions

This work provides both practical and theoretical contributions to the field of AI-driven hardware design automation:

• Practical Implications: With enhanced evaluation methodologies, there is better clarity on how to advance LLM capabilities for hardware description, potentially optimizing digital design processes and reducing time-to-market for hardware products.
• Theoretical Insights: The detailed failure analysis opens new avenues for understanding model architectures and the common pitfalls encountered in syntax-constrained generation tasks.

Looking forward, the paper suggests expanding benchmark tasks to encompass more diverse design flow elements, including verification and testbench generation, thereby advancing the integration of AI into all facets of digital hardware design. The exploration of larger and more diverse datasets, as well as improved model architectures, may significantly propel the proficiency of LLMs in hardware-related applications.