- The paper introduces a deterministic mutation-testing audit and finds that 46 of 60 self-certified testbenches pass a 95% simulation kill-rate floor, while correcting a false 100% result caused by a fail-open bug.
- Applying GateTruth to RTLLM v2.0 shows that 33 of 46 designs fall below the 95% floor, with a 56.8% pooled kill rate and three testbenches accepting mutants that invert primary outputs.
- The paper demonstrates that benchmark design choices affect conclusions: a 16,384-token cap substantially reordered model rankings, while limited artifact retention and protocol gaps weakened reproducibility.
Motivation and central claim
Benchmarks for evaluating LLMs on register-transfer-level (RTL) generation have converged on a common protocol: score pass@k against a testbench, treating the testbench as ground truth. The paper's central observation is that this assumption is almost never checked. A testbench with weak fault sensitivity passes a broken design as readily as a correct one, so a benchmark built on such testbenches silently overstates every model's performance in a way pass@k cannot reveal. The paper reports that, across every accessible RTL-generation benchmark paper, repository, and documentation set, none applies mutation testing — the established technique of injecting deterministic faults into a reference implementation and measuring the fraction a test suite detects — to its own testbenches. GateTruth addresses this gap with three artifacts: a mutation-audit methodology; a 68-task dual-track reference suite (60 specification-to-RTL tasks, 8 agentic-repair tasks) used to validate the methodology under fully controlled conditions; and an unmodified application of the same engine to RTLLM v2.0, a widely adopted external benchmark.
Audit methodology
The audit protocol is deterministic and seeded: a structural generator masks comments and string literals, enumerates every viable code site for a fixed operator set (comparator-boundary flips, operator/logic/bitwise inversions, shift-direction inversion, reset- and enable-polarity flips, assignment deletion and hold, output inversion), and executes mutants strictly sequentially, since verdicts are not invariant to concurrency — a mutant straddling its time budget may be classified as killed-by-timeout under parallel load but survive sequentially, letting parallel sweeps inflate kill rates. Timeouts are retried once at extended budget and then counted as indeterminate against the kill rate rather than dropped. A mandatory baseline validation requires the unmodified reference to pass the benchmark's own testbench before any mutant is generated; designs failing this bar are reported as unsupported, never scored as a misleading 0%. Build-convention normalization is limited to what a vendor's own build recipe documents, with every applied module-name alias recorded per design.
Because a tool auditing others' rigor must first demonstrate its own, the paper certifies its reference suite and discloses two successive errors found by turning the methodology on itself. First, the mutant generator carried hand-authored equivalent-mutant exclusions for 12 tasks; testing the excluded mutants directly against the real hidden testbenches killed 36 of 72 outright, falsifying the equivalence reasoning for half. Removing the mechanism moved the certified count from a false 60/60 to 49/60. Second, adversarial review found the "kill rate" blended three signals — lint failures, simulation kills, and formal-property kills — while the methodology claims to measure the simulation testbench specifically, plus a fail-open bug in which a missing hidden-vector mount caused every mutant to fail identically and be misclassified as killed (reproduced as a false 100% with exit code 0). Under the corrected engine, counting only behavioral simulation failures: 46 of 60 Track A testbenches clear the 95% floor, from 1,149 valid mutants (1,245 generated minus 96 stillborn), with a pooled simulation kill rate of 96.17%. The 14 below-floor testbenches result from three distinct causes: a Goodhart effect on ten testbenches revised specifically to pass the gate (seven of which still fail it today, partly because the revision consisted of the unsound exclusion mechanism), the exclusion defect itself, and formal-only kills silently blended into the metric. Notably, seven of the ten revised testbenches had a pre-revision median kill rate of 77.5% — close to RTLLM's measured median — though the authors concede the comparison is not fully like-for-like, since most unrevised testbenches were still authored inside a project aware of the gate.
External audit of RTLLM v2.0
Pointing the same engine at RTLLM v2.0 (vendor commit pinned, Icarus Verilog 12.0, -g2012), 46 of 50 designs passed baseline validation (26 requiring a documented compile-time alias). The findings are stark:
| Metric |
Value |
| Designs audited |
46 |
| Median per-design kill rate |
74.0% |
| Pooled kill rate |
56.8% |
| Designs below the 95% floor |
33 of 46 (72%) |
| Designs at exactly 100% |
13 |
| Designs at 0% |
3 |
Three designs (adder_8bit, edge_detect, square_wave) score 0%. Inspection of individual survivors confirms severity where denominators are small: square_wave's survivors include two inverted combinational outputs and adder_8bit's single mutant inverts its sum output — a testbench accepting an inverted primary output measures nothing about that output. Conversely, edge_detect's 0% rests on ten instances of a single operator (assignment_hold), establishing only insensitivity to register-hold faults. The below-floor finding is not a thin-denominator artifact: kill rate correlates negatively with mutant count (Spearman ρ=−0.39), with the largest-design tercile scoring worst (61.4% median). A determinism check re-ran a seeded nine-design sample byte-identically. One excluded design, radix2_div, fails its own shipped testbench outright independent of mutation, with definitively wrong quotients (e.g., $123/123$ yielding 6400) reproduced under both -g2001 and -g2012 — though only under a single simulator. For NVIDIA's CVDP, no audit was possible: the public release withholds golden solutions for anti-contamination reasons, removing the artifact mutation testing requires; the paper reports this as a structural auditability finding rather than omitting the benchmark.
The authors flag the key asymmetry themselves: their testbenches were authored under the gate being measured, while RTLLM's were not, so the comparison risks contrasting a tuned quantity with a held-out one. They bound this exposure by identifying 43 of 60 testbenches never touched by any of the three defect mechanisms that clear the floor as originally authored.
Reference-suite results and the token-budget finding
On Track A, evaluated at a 16,384-token cap, GPT-5 leads (pass@1 43/60, GateTruth Score 48.83), followed by Opus (42/60, 46.99) and Sonnet (38/60, 42.09); the GPT–Opus gap of 1.84 points is within Opus's measured run-to-run standard deviation (σ≈1.85), so first place is not claimed as resolved. Two structural results stand out. First, the leaderboard is overwhelmingly correctness-gate-driven: mean PPA over passed tasks spans only 3.1% across all seven models, and 142 of 235 passing pairs land within 10−6 of a PPA ratio of 1.000 — for simple Tier-1 tasks, any correct RTL synthesizes to near-identical netlists, so PPA discriminates mainly in agentic repair. Second, on Track B (agentic repair under strict budgets and sequential-equivalence gating), performance degrades sharply: Opus clears 5/8 objectives, Sonnet 3/8, GPT-5 1/8, and four models clear none. All three behavior-preserving, sec-gated objectives went unsolved by every model — though the authors note their sec implementation (induction depth 20, no retiming support, 60-second timeout prone to multiplier blow-up) restricts the legal solution space, and four sec timeouts leave those candidates' timing outcomes unmeasured, so the honest claim is that no model produced a solution the gate accepts, not that none exists.
A second methodological result emerged from self-auditing: the first campaign's uniform 4096-token output cap caused three models to return no parseable code on up to 22 of 60 tasks. Re-running at 16,384 tokens moved GPT-5 from fifth place to first (+17.19 points) and Gemini from last to sixth (+13.23). The authors explicitly decline to call this a controlled single-variable experiment — hosted model aliases were not pinned to immutable snapshots, the scoring-image build marker changed, and the response parser was hardened between runs — but the narrow claim stands: an output-token cap is a plausible experimental variable capable of reordering a leaderboard, and benchmarks that do not report their cap have not reported a comparable experimental condition.
Limitations and open questions
The paper is unusually explicit about validity threats. Sensitivity is not correctness: high kill rates do not establish oracle soundness or testbench specificity (false-rejection rate is unmeasured). No equivalent-mutant detection is performed, making reported rates conservative lower bounds. The audit covers a fixed generic operator set; survival rates vary sharply by operator (80% of bitwise inversions survive versus 0% of reset-polarity flips), so results reflect sensitivity to that distribution, not benchmark-independent quality. VerilogEval and several other benchmarks remain unaudited. Physical metrics are pre-layout estimates at a single corner, not signoff values. Four Track B protocol gaps were found only after the reported campaign: an unenforced read boundary (five successful testbench reads across two models, though never reaching hidden vectors), a discoverability gap leaving 38 of 56 runs with zero successful file reads, a token-budget check that 31 runs exceeded, and — most seriously — non-retention of all 56 submitted design files, making every Track B result permanently unreproducible despite the harness's recorded figures. The lint gate also materially shapes rankings: 99 of 132 lint failures are width-truncation warnings without hard errors, varying 2.4× across models, making Track A partly a joint ranking of correctness and width-declaration hygiene. Open questions left concrete include whether the three unsolved behavior-preserving objectives admit any solution passing both gates jointly, how many width-only lint failures would pass simulation under waivers, and whether the 14 sub-floor testbenches can be strengthened without introducing new defects.
Conclusion
GateTruth demonstrates that mutation-kill certification is feasible, cheap enough to run, and revealing: 72% of audited RTLLM v2.0 testbenches fall below a 95% kill floor, three miss inverted primary outputs entirely, and one benchmark ships a golden reference that fails its own testbench. Equally instructive is the self-audit, which found the same failure modes in the authors' own instrument — including a false 100% kill rate from a fail-open bug — and surfaced the output-token budget as a leaderboard-reordering experimental variable. The paper argues that mutation-kill certification should become a standard reporting requirement for RTL-generation benchmarks, and its own disclosure of 14 uncertified testbenches, unreproducible Track B manifests, and confounded reruns models the reporting standard it proposes.