---
title: 'Struct-Bench: Deterministic RTL Verification Framework'
url: https://www.emergentmind.com/topics/struct-bench
type: topic
---

# Struct-Bench: Deterministic RTL Verification Framework

Struct-Bench, denoted STG in its underlying formulation, is a template-driven, structure-aware framework for generating RTL testbenches deterministically in large language model (LLM)-driven HDL design workflows. It is designed for settings in which large numbers of candidate DUTs must be verified rapidly and reliably against a known golden reference, or, in an extension, a “silver” reference. Rather than treating testbench creation as unconstrained LLM code synthesis, Struct-Bench exploits the inherent structure of the DUT to analyze module structure, classify design type, emit a parameterized testbench shell, and drive stimuli according to a provably complete or high-coverage strategy. In the reported evaluation, this yields deterministic outputs, removes LLM token cost at test time, and supports direct verification, verification-oriented data curation, and test-time scaling workflows [2606.12983].

## 1. Problem setting and operational scope

Automated testbench generation is presented as a critical bottleneck in LLM-driven RTL workflows. The motivating observation is that existing prompt-based approaches treat testbench generation as unconstrained code synthesis, which leads to stochastic outputs, high token cost, low reproducibility, and insufficient coverage. Struct-Bench is introduced specifically to replace that regime with a deterministic verification backbone built from fixed templates plus small amounts of extracted design metadata [2606.12983].

The framework assumes a DUT together with a known golden reference, or alternatively a “silver” reference. Its scope is testbench generation and verification rather than RTL synthesis. A common misunderstanding is to treat it as another prompting method for code generation; the paper instead defines it as a structure-aware system whose outputs are rendered from templates after module parsing, design-type classification, signal-role classification, and strategy selection. This distinction matters because the framework’s main claim is not improved prompt engineering, but a change in the verification formulation: stimulus generation becomes a bounded structural procedure rather than free-form synthesis.

The reported workflow is explicitly dual-use. As a direct verification tool, Struct-Bench compares DUT and reference side by side under deterministic stimuli. As a data curation engine, it filters and verifies large corpora of generated RTL. As a test-time scaling oracle, it guides search procedures by supplying fast, structured verification feedback. These three uses are central to its identity.

## 2. Architecture and formal structural model

Struct-Bench is organized into three stages: Module Parsing & Analysis, Test Scenario Generator, and Structured Template Engine & Simulation Harness. In the parsing stage, the design parser invokes Icarus Verilog to build the module-instantiation DAG, identifies the top module as the root, and extracts port lists. A design-type classifier checks for clock/reset ports to distinguish combinational from sequential designs, and uses pattern matching or LLM assistance to detect FSMs. A signal-role classifier then assigns ports to one of $\{\text{clock}, \text{reset}, \text{control}, \text{data}\}$ via LCS-based fuzzy matching against hint lists such as “clk,” “rst,” “en,” and “addr,” together with simple width heuristics [2606.12983].

The module-instantiation structure is formalized as a directed acyclic graph
$$
G_{\mathrm{mod}}=(M,E),
$$
where $M$ denotes module definitions and there is an edge $m \to m'$ if module $m'$ is instantiated by module $m$. The unique root $m_0$ with the largest descendant count is taken as the top-level DUT. This representation provides the structural basis for identifying the verification target and for extracting the metadata required by the templating stage.

For FSM-dominated designs, Struct-Bench also constructs an FSM state-transition graph
$$
G_{\mathrm{FSM}}=(S,T), \qquad T \subseteq S \times S,
$$
either by pattern matching on constructs such as `always @(posedge clk) case (state) ... endcase` or by querying an LLM with a structured prompt to extract a JSON description. Each transition guard is decomposed into a drivable component and a waiting component: the former ranges over drivable ports such as `req=1`, and the latter ranges over internal conditions such as `count>=3`. In the C++ harness compiled via Verilator, each guard is converted into concrete port assignments followed by clock ticks until the wait condition is met or a timeout occurs; successful traversals record coverage and recurse depth-first, while failures backtrack.

The final stage uses Jinja templates for Verilog, or C++ via Verilator, to render a testbench shell that instantiates DUT and reference side by side, declares input drivers, builds clock/reset generators, and inserts an output-comparator block that increments per-port error counters. The rendered testbench is then compiled by Verilator or a standard Verilog toolchain, simulated, and instrumented for pass/fail statistics, line coverage, toggle coverage, and, for FSMs, transition coverage. Because the logic is template-generated from fixed metadata, the output is deterministic across runs.

## 3. Stimulus generation and coverage semantics

Struct-Bench defines three stimulus strategies, selected by design type. For combinational designs, it enumerates all control vectors
$$
c \in \{0,\dots,2^{b_c}-1\},
$$
and, for each such vector, samples $N_s$ random data vectors
$$
d \sim \mathrm{Uniform}(\{0,\dots,2^{b_d}-1\}).
$$
The total number of test vectors is therefore $2^{b_c}\times N_s$, with the bound
$$
2^{b_c}\times N_s \le 2^{b_{\max}}
$$
enforced by reducing $N_s$ as needed. After each $(c,d)$ pair, the framework immediately invokes the reference and DUT, compares all outputs, and increments error counters. The associated coverage model includes control-mode coverage
$$
C_c = \frac{\#\text{control vectors tested}}{2^{b_c}},
$$
while the data-sampling component yields a probability
$$
1-(1-p)^{N_s}
$$
of hitting corner data cases when $p$ denotes the density of “interesting” data patterns [2606.12983].

For general sequential designs, the stimulus schedule is randomized but still structurally fixed. Pass I omits resets: it drives random data for $N_1$ cycles, then enumerates control vectors $c$, holding each $c$ for $L$ cycles while driving new random data each cycle. Pass II incorporates resets: between control cycles, reset is asserted and deasserted, synchronously or asynchronously according to the port classification. Stimuli are injected at multiple phases within the cycle—specifically intra-cycle, posedge, and negedge—and comparison is performed at each phase in order to catch Mealy versus Moore outputs. The reported coverage metrics are
$$
\mathrm{LineCov} = \frac{\#\text{lines executed at least once}}{\#\text{total lines}}
$$
and
$$
\mathrm{ToggleCov} = \frac{\#\text{signals toggled at least once}}{\#\text{total signals}}.
$$

For FSM-guided designs, Struct-Bench performs a DFS traversal of $G_{\mathrm{FSM}}$, exercising each transition exactly once barring infeasible paths. Runtime per-transition coverage is confirmed by instrumenting coverage counters on the case statements associated with each guard, and the result is reported as 100% transition coverage. This is the strongest of the three strategies in terms of structural completeness, but only for the transition graph that has been extracted; a plausible implication is that the quality of FSM extraction directly bounds the quality of the resulting verification.

## 4. Measured performance and verification behavior

The direct comparison reported on the VerilogEval benchmark covers 156 problems and 3,046 DUT variants. Relative to a pure-LLM testbench flow, Struct-Bench reduces generation time from 92.4 s to 0.13 s, increases line coverage from 93.97% to 95.88%, and increases toggle coverage from 85.40% to 95.77% [2606.12983].

| Metric | Pure-LLM testbench | STG |
|---|---:|---:|
| Generation time | 92.4 s | 0.13 s |
| Line coverage | 93.97% | 95.88% |
| Toggle coverage | 85.40% | 95.77% |

The DUT classification agreement analysis across 3,046 cases further separates agreement from disagreement modes. Both methods were correct in 2,734 cases, corresponding to 89.8%. “LLM PASS, STG FAIL (false-fail)” occurred in 27 cases, or 0.9%. “LLM FAIL, STG PASS (false-pass)” occurred in 236 cases, or 7.8%. Both failed in 49 cases, or 1.6%. The paper also states that Struct-Bench helps identify errors in RTL generation benchmarks by exposing faulty benchmark testbenches, which suggests that part of its empirical value lies not only in evaluating DUTs but also in auditing the evaluation infrastructure itself.

In a large-scale data-curation run over 115,000 problems, Struct-Bench is reported as 11x faster than LLM-based filtering on a single CPU core with 127x less energy. The detailed figures are runtime 59.1 h versus 5.6 h, power draw 1,200 W versus approximately 100 W, total energy 70.9 kWh versus 0.56 kWh, and compilation success 71.3% versus 100%. These measurements position the framework as a batch verification system rather than merely a convenience layer on top of prompting.

## 5. Case studies, data curation, and search guidance

A representative case study is the Deep-State Mealy Detector with 15 states. Under random sequential stimulus, state-visit counts decay exponentially and states $S_{11}\dots S_{14}$ are never reached. By contrast, FSM-guided traversal achieves 100% transition coverage in one pass and exposes deep-state bugs that random methods miss [2606.12983]. This example illustrates the framework’s central thesis: once the design structure is made explicit, stimulus generation can target latent verification gaps that stochastic random schedules systematically undersample.

The same paper describes a PyraNet data-curation pipeline that scales from 692k samples to 115k selected samples and finally to 43k verified samples. The procedure has three steps. First, easy problems solvable by small base models are filtered out via Struct-Bench pass/fail. Second, a teacher LLM generates DUT and trace, and Struct-Bench verifies them against the golden reference. Third, only correct DUTs, totaling 43k, are retained for SFT training. The downstream result is that three student models—Qwen2.5-Coder-7B, Qwen3-4B, and Qwen3-8B—trained with a single SFT stage on Struct-Bench-curated data match or outperform prior specialized models on VerilogEval, RTLLM, and CVDP, despite not using RL. The paper characterizes these distilled models as providing state-of-the-art performance in its multi-benchmark evaluation.

Struct-Bench is also used as a test-time scaling oracle in MCTS-based RTL refinement. When plugged into VFlow’s search loop, it reduces median search nodes by 14–47% across four backbone LLMs, while slightly improving final pass rates, with improvements of up to +1.5 percentage points at 256 nodes. This places Struct-Bench in a broader class of verifier-guided search systems, but its distinctive contribution is that the verification oracle is itself deterministic, template-based, and designed for HDL structure.

## 6. Integration constraints, limitations, and terminological ambiguity

The practical integration model is straightforward. For known-reference verification, the workflow requires a DUT and golden RTL, and Struct-Bench can be run in “automatic” mode on a CPU cluster without manual prompts, emitting compilable testbenches. For challenging modules, an interactive mode permits overrides of signal roles or design type hints. If no golden HDL is available, the silver-reference extension allows an LLM-generated C++ or SystemC model stub to serve as the oracle, with the caveat that oracle correctness then inherits LLM stochasticity. The parsing-and-render pipeline scales trivially across cores, while LLM calls for FSM extraction can be limited to ambiguous cases only [2606.12983].

The paper identifies four main limitations. First, FSM extraction still relies on heuristics or LLMs; very large or deeply pipelined state machines may require more robust static analysis than name-based pattern matching affords. Second, complex RTL features, particularly SystemVerilog-specific constructs such as interfaces and dynamic arrays, are not yet fully supported by the lightweight parser. Third, silver references are noisy when gold is unavailable, reintroducing stochastic verdicts. Fourth, coverage remains limited to line, toggle, and transition metrics; path or functional coverage, including branch or assertion coverage, is described as an open extension. A common misconception is therefore that Struct-Bench guarantees complete verification. The source only claims deterministic generation together with provably complete or high-coverage strategies under specific structural assumptions.

The term “Struct-Bench” is not unique in the arXiv literature. It also names a benchmark for differentially private structured text generation based on CFG-defined structure [2509.10696]. Closely related names include “LLMStructBench,” a benchmark for structured data extraction to JSON [2602.14743], and “StructFlowBench,” a benchmark for multi-turn instruction following with structural flow modeling [2502.14494]. In the HDL-verification context, however, Struct-Bench refers to the STG framework for deterministic RTL testbench generation and verification-oriented data curation, not to those other benchmark families.

Source: https://www.emergentmind.com/topics/struct-bench