---
title: 'CorrectBench: Self-Correcting HDL Testbenches'
url: https://www.emergentmind.com/topics/correctbench-suite
type: topic
---

# CorrectBench: Self-Correcting HDL Testbenches

CorrectBench is an automatic, large language model (LLM)-based framework for generating and functionally validating testbenches for hardware description language (HDL) designs, introducing functional self-validation and self-correction mechanisms to address the instability and low functional yield of purely generative LLM testbench methods. Given only a natural-language specification of the design-under-test (DUT), CorrectBench integrates guided testbench synthesis, scenario-driven validation, and an iterative, diagnosis-driven correction loop, yielding substantial gains in testbench functional correctness and robustness compared to prior LLM-driven approaches [2411.08510].

## 1. Architecture and Workflow

CorrectBench operates as a closed-loop framework comprising four interacting components:

1. **Testbench Generator (F₉):** Utilizes AutoBench to convert the natural-language DUT specification (SPEC) into an initial ("raw") hybrid testbench, consisting of:
   - A Verilog driver ("scenarios"), where each scenario comprises named stimulus sequences
   - A Python checker for output comparison against reference models

2. **Scenario-Based Functional Validator (Fᵥ):** Given SPEC and the raw testbench, produces $N_r = 20$ candidate Verilog RTL implementations (via LLM). Each candidate is simulated against all $N_s$ scenarios, recording outcomes as a Boolean correctness matrix ("RS matrix", $R \in \{0,1\}^{N_r \times N_s}$).

3. **LLM-Based Corrector (F꜀):** Upon validation failure, this module receives the SPEC, current testbench, scenario listings, and scenario indexes identified as erroneous, engaging in a chain-of-thought reasoning process to:
   - Diagnose root cause ("why")
   - Locate the corresponding driver code defect ("where")
   - Propose and generate code fixes ("how")

4. **Action Agent:** Orchestrates the workflow by alternately invoking validation, correction (maximum 3 per testbench), or generator reboots (maximum 10 per spec), according to the following pseudo-coded decision logic:

   ```
   TB ← F₉(SPEC)
   I_C ← I_R ← 0
   while True do
     (C_TB, Bugs) ← Fᵥ(TB)
     if C_TB = false and I_C < I_C^max then
       I_C ← I_C + 1
       TB ← F꜀(TB, Bugs)
     else if C_TB = false and I_R < I_R^max then
       I_R ← I_R + 1; I_C ← 0
       TB ← F₉(SPEC)
     else
       break
   TB₍final₎ ← TB
   ```

## 2. Algorithms and Prompt Strategies

### 2.1 Natural-Language RTL Processing (Generation)

The LLM (e.g., gpt-4o) is prompted with SPEC and an instruction template to generate:
- Verilog driver skeleton
- List of named test scenarios (detailed in natural language)
- Python checker template

The approach incorporates syntax debugging, code and scenario completion refinements as in AutoBench.

### 2.2 Scenario-Based Functional Self-Validation

For the raw testbench, the validator:
- Generates $N_r = 20$ candidate RTLs ("Generate a Verilog RTL implementation consistent with SPEC")
- Simulates each candidate ($\mathrm{RTL}_i$) against each scenario ($j$)
- Records pass ($R_{i,j}=1$) if outputs match Python checker; $0$ otherwise
- Forms the RS matrix

**Validation criterion ("70%-wrong + row-exception"):**
- Scenario $j$ is 'wrong' if $\geq 70\%$ of candidates fail ($R_{*,j}=0$)
- If any row is 'entirely green' ($R_{i,*}=1$) in more than $25\%$ of rows, the testbench is accepted as correct
- Otherwise, record indexes of wrong scenarios and mark the testbench as incorrect

### 2.3 LLM-Based Corrector

**Stage 1 (Reasoning):** Chain-of-thought prompts direct the LLM to address:
- "Why are these scenario outputs incorrect?"
- "Where in the Verilog driver code does this mistake appear?"
- "How should we change the code or stimulus definitions to fix it?"

**Stage 2 (Code Repair):** A code-injection prompt presents the driver skeleton and requests the LLM to output only the corrected core logic, preserving boilerplate code.

## 3. Experimental Setup and Evaluation Metrics

- **Dataset:** 156 HDLBits problems (81 combinational, 75 sequential)
- **LLMs Tested:** gpt-4o-2024-08-06 (primary), claude-3.5-sonnet, gpt-4o-mini (additional)
- **Simulator:** Icarus Verilog with Python 3.12
- **Evaluation Protocol (AutoEval):**
  - $\mathrm{Eval}_0$: syntactically correct and executable testbench
  - $\mathrm{Eval}_1$: testbench passes golden RTL in a single run
  - $\mathrm{Eval}_2$ (primary): passes or fails match in $\geq 80\%$ of 10 mutated RTLs per task
- **Primary Metrics:** Pass ratios (%) under $\mathrm{Eval}_2$ and average number of passed tasks

## 4. Results and Comparative Performance

### Main Pass Ratios (Eval₂, 156 tasks)

| Method                | Total   | Comb.   | Seq.    |
|-----------------------|---------|---------|---------|
| Baseline (direct LLM) | 33.33%  | 53.58%  | 11.47%  |
| AutoBench             | 52.18%  | 69.14%  | 33.87%  |
| CorrectBench          | 70.13%  | 84.20%  | 54.93%  |

- **Relative Gains:** Over AutoBench, CorrectBench improves pass ratios by $+36.8\%$ (total), $+30.6\%$ (combinational), and $+43.5\%$ (sequential).
- **Component Contributions:** The validator alone adds $\sim26.8$ passed tasks relative to AutoBench; the corrector provides an additional $\sim9.2$ tasks (34.3% of validator’s gain).
- **Validator Accuracy:** On 1,560 held-out TBs, correct-TB accuracy is $96.7\%$, wrong-TB $81.0\%$, and overall $88.85\%$ under the 70%-wrong criterion.
- **Ablation Thresholds:** Stricter thresholds (e.g., 100%-wrong) yield lower validator accuracy and increased false alarms, while the chosen 70% threshold balances validation quality and correction frequency.
- **Portability:** Consistent hierarchy (Baseline < AutoBench < CorrectBench) across tested LLMs.

## 5. Rationale, Limitations, and Methodological Significance

Self-validation transforms generative model hallucinations into actionable, scenario-localized error reports without human oracles, facilitating more precise automated corrections. The RS matrix enables localized diagnosis by pinpointing erroneous scenarios, avoiding all-or-nothing feedback that impedes correction. The two-stage corrector employs explicit, structured chain-of-thought prompting for diagnosis and repair, producing more reliable testbench patches than naïve regeneration.

**Identified limitations:**
- Validation thresholds are empirically tuned; adaptive or coverage-based thresholds could enhance robustness.
- Simulation cost is increased by the requirement to generate and evaluate 20 RTL candidate implementations per validation.
- Certain highly complex modules may require advanced metrics beyond simple pass/fail to capture behavioral subtleties.

## 6. Applications and Future Extensions

Potential extensions of CorrectBench include:
- Integration of structural or formal coverage metrics into the RS matrix for coverage-driven testbench generation
- Hierarchical scenario definitions and distributed correction strategies for scaling to full IP blocks
- Automatic SPEC extraction from C/C++ for seamless HLS-to-RTL and testbench generation
- Formal verification augmentation using validator-discovered counterexamples to seed assertion or constraint synthesis

In summary, CorrectBench achieves substantial increases in the functional robustness of LLM-generated HDL testbenches, marking a significant advance by enabling effective self-validation and self-correction without human reference designs or oracles [2411.08510].

Source: https://www.emergentmind.com/topics/correctbench-suite