CorrectBench: Self-Correcting HDL Testbenches

• CorrectBench is an LLM-based framework that automatically generates and functionally validates HDL testbenches from natural-language specifications.
• It integrates guided testbench synthesis, scenario-driven validation, and an iterative correction loop to diagnose and fix design defects.
• By achieving a 70.13% pass ratio, CorrectBench significantly outperforms baseline methods in enhancing testbench robustness and accuracy.

CorrectBench is an automatic, LLM-based framework for generating and functionally validating testbenches for hardware description language (HDL) designs, introducing functional self-validation and self-correction mechanisms to address the instability and low functional yield of purely generative LLM testbench methods. Given only a natural-language specification of the design-under-test (DUT), CorrectBench integrates guided testbench synthesis, scenario-driven validation, and an iterative, diagnosis-driven correction loop, yielding substantial gains in testbench functional correctness and robustness compared to prior LLM-driven approaches (Qiu et al., 2024).

1. Architecture and Workflow

CorrectBench operates as a closed-loop framework comprising four interacting components:

1. Testbench Generator (F₉): Utilizes AutoBench to convert the natural-language DUT specification (SPEC) into an initial ("raw") hybrid testbench, consisting of:
   • A Verilog driver ("scenarios"), where each scenario comprises named stimulus sequences
   • A Python checker for output comparison against reference models
2. Scenario-Based Functional Validator (Fᵥ): Given SPEC and the raw testbench, produces $N_r = 20$ candidate Verilog RTL implementations (via LLM). Each candidate is simulated against all $N_s$ scenarios, recording outcomes as a Boolean correctness matrix ("RS matrix", $R \in \{0,1\}^{N_r \times N_s}$).
3. LLM-Based Corrector (F꜀): Upon validation failure, this module receives the SPEC, current testbench, scenario listings, and scenario indexes identified as erroneous, engaging in a chain-of-thought reasoning process to:
   • Diagnose root cause ("why")
   • Locate the corresponding driver code defect ("where")
   • Propose and generate code fixes ("how")
4. Action Agent: Orchestrates the workflow by alternately invoking validation, correction (maximum 3 per testbench), or generator reboots (maximum 10 per spec), according to the following pseudo-coded decision logic:

$R \in \{0,1\}^{N_r \times N_s}$6

2. Algorithms and Prompt Strategies

2.1 Natural-Language RTL Processing (Generation)

The LLM (e.g., gpt-4o) is prompted with SPEC and an instruction template to generate:

• Verilog driver skeleton
• List of named test scenarios (detailed in natural language)
• Python checker template

The approach incorporates syntax debugging, code and scenario completion refinements as in AutoBench.

2.2 Scenario-Based Functional Self-Validation

For the raw testbench, the validator:

• Generates $N_r = 20$ candidate RTLs ("Generate a Verilog RTL implementation consistent with SPEC")
• Simulates each candidate ($\mathrm{RTL}_i$) against each scenario ($j$)
• Records pass ($R_{i,j}=1$) if outputs match Python checker; $0$ otherwise
• Forms the RS matrix

Validation criterion ("70%-wrong + row-exception"):

• Scenario $j$ is 'wrong' if $\geq 70\%$ of candidates fail ($N_s$0)
• If any row is 'entirely green' ($N_s$1) in more than $N_s$2 of rows, the testbench is accepted as correct
• Otherwise, record indexes of wrong scenarios and mark the testbench as incorrect

2.3 LLM-Based Corrector

Stage 1 (Reasoning): Chain-of-thought prompts direct the LLM to address:

• "Why are these scenario outputs incorrect?"
• "Where in the Verilog driver code does this mistake appear?"
• "How should we change the code or stimulus definitions to fix it?"

Stage 2 (Code Repair): A code-injection prompt presents the driver skeleton and requests the LLM to output only the corrected core logic, preserving boilerplate code.

3. Experimental Setup and Evaluation Metrics

• Dataset: 156 HDLBits problems (81 combinational, 75 sequential)
• LLMs Tested: gpt-4o-2024-08-06 (primary), claude-3.5-sonnet, gpt-4o-mini (additional)
• Simulator: Icarus Verilog with Python 3.12
• Evaluation Protocol (AutoEval):
  • $N_s$3: syntactically correct and executable testbench
  • $N_s$4: testbench passes golden RTL in a single run
  • $N_s$5 (primary): passes or fails match in $N_s$6 of 10 mutated RTLs per task
• Primary Metrics: Pass ratios (%) under $N_s$7 and average number of passed tasks

4. Results and Comparative Performance

Main Pass Ratios (Eval₂, 156 tasks)

Method	Total	Comb.	Seq.
Baseline (direct LLM)	33.33%	53.58%	11.47%
AutoBench	52.18%	69.14%	33.87%
CorrectBench	70.13%	84.20%	54.93%

• Relative Gains: Over AutoBench, CorrectBench improves pass ratios by $N_s$8 (total), $N_s$9 (combinational), and $R \in \{0,1\}^{N_r \times N_s}$0 (sequential).
• Component Contributions: The validator alone adds $R \in \{0,1\}^{N_r \times N_s}$1 passed tasks relative to AutoBench; the corrector provides an additional $R \in \{0,1\}^{N_r \times N_s}$2 tasks (34.3% of validator’s gain).
• Validator Accuracy: On 1,560 held-out TBs, correct-TB accuracy is $R \in \{0,1\}^{N_r \times N_s}$3, wrong-TB $R \in \{0,1\}^{N_r \times N_s}$4, and overall $R \in \{0,1\}^{N_r \times N_s}$5 under the 70%-wrong criterion.
• Ablation Thresholds: Stricter thresholds (e.g., 100%-wrong) yield lower validator accuracy and increased false alarms, while the chosen 70% threshold balances validation quality and correction frequency.
• Portability: Consistent hierarchy (Baseline < AutoBench < CorrectBench) across tested LLMs.

5. Rationale, Limitations, and Methodological Significance

Self-validation transforms generative model hallucinations into actionable, scenario-localized error reports without human oracles, facilitating more precise automated corrections. The RS matrix enables localized diagnosis by pinpointing erroneous scenarios, avoiding all-or-nothing feedback that impedes correction. The two-stage corrector employs explicit, structured chain-of-thought prompting for diagnosis and repair, producing more reliable testbench patches than naïve regeneration.

Identified limitations:

• Validation thresholds are empirically tuned; adaptive or coverage-based thresholds could enhance robustness.
• Simulation cost is increased by the requirement to generate and evaluate 20 RTL candidate implementations per validation.
• Certain highly complex modules may require advanced metrics beyond simple pass/fail to capture behavioral subtleties.

6. Applications and Future Extensions

Potential extensions of CorrectBench include:

• Integration of structural or formal coverage metrics into the RS matrix for coverage-driven testbench generation
• Hierarchical scenario definitions and distributed correction strategies for scaling to full IP blocks
• Automatic SPEC extraction from C/C++ for seamless HLS-to-RTL and testbench generation
• Formal verification augmentation using validator-discovered counterexamples to seed assertion or constraint synthesis

In summary, CorrectBench achieves substantial increases in the functional robustness of LLM-generated HDL testbenches, marking a significant advance by enabling effective self-validation and self-correction without human reference designs or oracles (Qiu et al., 2024).