Closing the Loop on LLM-Generated RTL Assertions with Quality-Aware Formal Verification

Abstract: LLM based assertion generation is making formal verification more accessible for Register Transfer Level (RTL) designs, but three practical issues remain. Generated properties can pass for the wrong reason, proof cost can vary widely from one design to another, and failing traces are hard to interpret. This paper presents a lightweight, open-source framework that addresses these issues in one loop. Our method combines mutation-guided refinement to reject weak assertions, including vacuous ones and those that fail to distinguish faulty behaviour, a solver-selection stage that chooses among candidate Satisfiability Modulo Theories (SMT) backends using RTL structure, and causal narrative synthesis to explain why a proof failed. Across diverse RTL designs, the framework improves confidence in generated assertions, reduces runtime variability over fixed-solver choices, and produces failure explanations that remain grounded in the counterexample trace. The results suggest that quality-aware closure, rather than assertion generation alone, is the missing step for practical LLM-assisted formal verification.

• The paper introduces a framework that closes the loop on LLM-generated RTL assertions by integrating mutation-guided refinement and solver selection.
• It achieves 90–100% effective kill rate in most designs, reducing mean solver latency by ~49% and enhancing diagnostic trace interpretability.
• The approach combines adversarial testing with causal narrative synthesis to eliminate vacuous assertions and provide clear debugging insights.

Quality-Aware Closed Loop for LLM-Generated RTL Assertion Formal Verification

Motivation and Problem Statement

The integration of LLMs in formal verification for RTL hardware accelerates the generation of verification assertions, but quality assurance remains problematic. Common issues include vacuous property satisfaction, inconsistent solver performance across designs, and indecipherable counterexample traces during assertion failure. This paper addresses these bottlenecks through a lightweight, open-source framework that closes the loop on assertion quality and practical verification, focusing not just on property synthesis but also meaningfulness, solver suitability, and interpretability of failure traces.

Framework Architecture

The proposed framework orchestrates four interconnected stages for quality control:

1. Initial Harness Generation: Produces a baseline set of assertions, assumptions, and helper logic using LLMs, bootstrapped from RTL and specification input.
2. Architecture-Aware Solver Selection: Profiles RTL structure before translating to SMT formulae, computes a structural score ($\chi$) considering datapath width, arithmetic density, mux density, memory size, and index complexity, then selects the optimal SMT solver backend. Unlike prior ML-based selector approaches operating on derived netlists [machsmt] [btor2select], this solution operates directly on raw RTL.
3. Mutation-Guided Refinement (MGR): Injects synthetic RTL faults (mutants), audits harness discriminability by effective kill rate, and iteratively refines assertion sets. Assertions are rejected if vacuous or incapable of distinguishing faulty behavior, with survivors fed back for harness strengthening.
4. Causal Narrative Synthesis (CNS): Summarizes proof failures by extracting localized triggering events and signal transitions from counterexample traces and synthesizing concise, faithful natural-language explanations via LLMs, anchored to the actual trace with no hallucination.

Mutation-Guided Refinement: Adversarial Assertion Quality

MGR operationalizes adversarial assertion refinement via mutation testing [mutation_jia], using the effective kill rate ($\rho_{\mathrm{eff}}$) as metric. Mutants are classified as Killed, Survived, Invalid, or Timeout. Harness refinement proceeds until $\rho_{\mathrm{eff}} \geq T$ (practically set to 85%). Notably:

• Strong numerical results: Across seven RTL designs, 90–100% effective kill rate is achieved in most cases except SDRAM, where timing mutations remain undetectable within bounded proof window. This plateau effect highlights the theoretical limitation of observable behavior within bounded model checking depth.
• Distinct handling: Surviving mutants resulting from structural or timing edits unobservable in the proof are not treated as evidence of assertion weakness, addressing a frequent issue where coverage metrics are misinterpreted.

Solver Selection: Runtime Stability via RTL Profiling

Solver selection is guided by structural metrics rather than post-translation formula properties. The structural score incorporates architectural features directly relevant to backend SMT solver behavior. Empirical results:

• Contradictory to default heuristics: The selector matches oracle-best choices, reducing mean step latency by ~49% compared to fixed (Yices-only) policies and avoiding undesirable timeouts observed with Z3 for certain wide-datapath or memory-dense designs.
• Hardware-centric interpretability: Uses dimension-reduced metrics interpretable by hardware engineers, countering the black-box nature of ML-based formula selectors.

Causal Narrative Synthesis: Trace Faithfulness

CNS evaluation considers debug faithfulness: cycle identification, signal relevance, mechanism classification, and absence of unsupported signal claims. Experimental results in 8 failure cases show:

• Baseline prompt: 37.5% overall faithfulness; signals always correct, cycle/mechanism often misclassified.
• Refined prompt: Increases faithfulness to at least 75%; with hints for challenging cases, reaches 100%.
• Signal identification is robust; claim hallucination is eliminated. Explanations remain concise, lucid, and actionable.

Implications

Practical implications: The framework materially increases trustworthiness and accessibility of LLM-assisted FV by automating quality assurance, making assertion generation non-vacuous, stabilizing solver runtime, and replacing opaque traces with actionable causal narratives.

Theoretical implications: MGR operationalizes adversarial closure, enabling richer coverage assurance than conventional metrics, and demonstrates the infeasibility of full observation within bounded window for certain mutation classes. The architecture-aware solver selection foregrounds hardware-specific design features as predictive for backend performance, challenging model-based selection paradigms.

Future developments: Extending MGR with explicit coverage metrics, more granular solver-selection baselines, and CNS evaluation on real regression traces (rather than mutation-induced failures) will further generalize and strengthen the framework. Integration into agentic formal workflows [saarthi24] [hey_ai] will enable more automated and interpretive formal verification pipelines.

Conclusion

This paper demonstrates that quality-aware closure—comprising mutation-guided refinement, architecture-aware solver selection, and trace-faithful causal narratives—is essential for practical LLM-assisted formal verification in hardware design. High assertion kill rates, stable runtime, and interpretive debugging collectively address major pain points in LLM-based workflows. The framework sets a robust precedent for integrating adversarial and explainable verification protocols into scalable FV pipelines, with significant potential for widespread adoption and further research (2606.21451).