FVDebug: LLM-Assisted RTL Debugging

• FVDebug is an LLM-driven debugging assistant that converts formal counterexamples into causal graphs for automated root cause analysis in RTL designs.
• It employs a four-stage pipeline—Graph Synthesis, Graph Scanner, Insight Rover, and Fix Generator—to generate ranked repair suggestions and diagnose design issues.
• Evaluations demonstrate high accuracy in root cause identification and fix generation, though challenges remain for complex processor-scale debugging.

Searching arXiv for FVDebug and closely related papers to ground the article. FVDebug is an LLM-driven debugging assistant for automated root cause analysis of formal verification failures in RTL hardware designs. It takes as input a failing formal property, the RTL design, and the formal counterexample trace, and it uses specifications and any available design documentation as auxiliary context. Its outputs are ranked root-cause hypotheses, a causal timeline explaining how the failure arose cycle by cycle, and concrete fix suggestions for the RTL or, in some cases, the formal environment such as missing constraints. The system is organized as a four-stage pipeline: Causal Graph Synthesis, Graph Scanner, Insight Rover, and Fix Generator (Bai et al., 16 Sep 2025).

1. Scope, problem setting, and core representation

FVDebug is motivated by the claim that debugging formal verification failures is one of the most time-consuming bottlenecks in modern hardware design workflows. The underlying paper states that current formal tools such as JasperGold or VC Formal are good at producing counterexamples, but that they largely stop at showing the engineer a waveform. In that account, the expensive step is the manual reasoning that follows: tracing through multi-cycle counterexamples, correlating waveform behavior with RTL, and reconciling implementation with design intent (Bai et al., 16 Sep 2025).

The system treats this as a reasoning problem over heterogeneous artifacts rather than as a theorem-proving problem alone. Waveforms provide dynamic evidence; RTL provides implementation semantics; specifications and design documentation provide intended behavior; and the failing property anchors both the initial failure event and the expected behavior. FVDebug’s distinctive move is to convert the counterexample into a causal graph rather than leaving it as a flat waveform.

The paper formalizes that graph as

$\mathcal{G} = (\mathcal{V}, \mathcal{E}),$

where $\mathcal{V}$ is the set of nodes and $\mathcal{E}$ the set of edges. A node is a signal event, conceptually a tuple of signal name, cycle, and value, described repeatedly as $(\text{signal}, \text{cycle}, \text{value})$ or “signal@cycle=value.” An edge indicates immediate causal dependency between events. The system objective is operational rather than expressed as a single optimization function: given a failing property, RTL, and counterexample trace, identify the root cause and generate fixes that make the design satisfy the property (Bai et al., 16 Sep 2025).

2. Causal Graph Synthesis

Causal Graph Synthesis is the structural foundation of FVDebug. Instead of serializing the waveform into a long textual trace, the system recursively asks JasperGold why each observed event happened, using JasperGold’s built-in visualize -why command. Starting from the failing property at the violation cycle, it queries the immediate causes of that failure event, adds edges from parent events to child events, and then recurses on the parents. The default recursion depth is 20 cycles, a heuristic chosen to trade off completeness and cost (Bai et al., 16 Sep 2025).

The initial result of this process is a tree rooted at the assertion failure. Because reconvergent fan-in causes duplicate appearances of the same event, FVDebug consolidates that tree into a DAG by merging duplicates. The appendix states that each node is uniquely identified by the tuple (signal_name, cycle, value). When multiple tree paths converge to the same signal event, they are merged into a single DAG node while preserving all incoming edges. The graph is therefore built deterministically from Jasper’s causal explanations, the depth cutoff, and the deduplication rule; the paper does not define a learned edge-weighting rule or probabilistic graph-construction model (Bai et al., 16 Sep 2025).

This representation is intended to compress waveform reasoning into a structure that matches causal analysis more directly. The counterexample provides the observed values over time, but causality itself comes from the formal engine’s visualize -why responses. RTL and specifications are attached later as explanatory context rather than as inputs to edge construction. A plausible implication is that FVDebug treats the formal engine as the authoritative local cause extractor and the LLM as the higher-level reasoner over that extracted structure.

3. Graph Scanner and Insight Rover

Graph Scanner is the first LLM stage. Its task is not to explain the whole failure, but to identify suspicious local events and annotate them. The scanner traverses the DAG level by level in topological order and uses token-aware batching. The paper gives the algorithm explicitly as “Algorithm 1: Graph Scanner with Token-Aware Batching,” in which each level is partitioned by a BinarySearchMaxBatch routine under a token budget, prompts are built with cached context, and the LLM returns one analysis per node. In the appendix, the scanner budget is stated as 50,000 tokens per prompt; single-node context is typically 800–1200 tokens; BinarySearchMaxBatch searches between 1 and $$\min(|\text{remaining nodes}|,\lfloor 50000/\text{single\_node\_tokens}\rfloor)$$; and actual batch sizes ranged from 3 to 8 depending on context complexity (Bai et al., 16 Sep 2025).

The scanner’s most distinctive prompt design is “for-and-against prompting.” For every node, the prompt requires arguments both FOR the node being suspicious and AGAINST it being suspicious. The prompt also requires exact file:line references, quoted RTL evidence, and an explicit statement if RTL context is missing. Its scoring rubric is:

• $0.9-1.0$: direct RTL bug found
• $0.7-0.8$: likely logic error
• $0.5-0.6$: suspicious pattern
• $0.3-0.4$: downstream symptom
• $0.0-0.2$: normal or insufficient evidence

The returned JSON fields are is_suspicious, is_key_event, suspicion_score, importance_score, causal_validity, and analysis. The paper presents this as a defense against confirmation bias: a node is not accepted merely because the waveform matches the implemented RTL; the prompt forces the model to ask whether the RTL itself makes semantic sense (Bai et al., 16 Sep 2025).

Insight Rover is the second LLM stage and converts suspicious local events into failure explanations. The paper formalizes a set of competing hypotheses $\mathcal{V}$0, with each $\mathcal{V}$1 maintaining a narrative description, chronological timeline, evidence collection, confidence score, and frontier of unexplored nodes. “Algorithm 2: Insight Rover: Agentic Hypothesis Exploration” starts by creating one hypothesis per suspicious node through LLM.GenerateTheory(node, \mathcal{G}), then repeatedly selects frontier nodes, analyzes them in context, updates the hypothesis, expands the frontier, manages the narrative pool, and finally ranks the resulting hypotheses with LLM.EvaluateAll(\mathcal{H}) (Bai et al., 16 Sep 2025).

The rover considers several classes of problems: RTL bug, under-constrained inputs, assertion/property issue, or design-intent mismatch. Its node-selection prompt is shown summaries of frontier nodes, including signal/value and graph properties such as in-degree and out-degree. Frontier size is capped at 20 nodes, prioritizing those with higher suspicion scores. The appendix states that the intelligent exploration examines about 15–20 nodes per hypothesis on average, versus more than 100 for breadth-first search to the same depth, reducing the search space by over 90%. The paper presents this as evidence that structured narrative exploration is doing more than exhaustive traversal (Bai et al., 16 Sep 2025).

The paper also contains a small implementation inconsistency. The main text says the system dynamically adjusts to accommodate all suspicious nodes and gives a minimum of 3 narratives, while the appendix says “FVDebug maintains exactly three active narratives throughout exploration,” then later says it automatically expands to accommodate all suspicious nodes. A second inconsistency concerns final hypothesis ranking: the appendix reports explicit criterion weights—causal sufficiency 0.3, evidence quality 0.25, mechanistic clarity 0.25, actionability 0.15, narrative coherence 0.05—whereas the ranking prompt says overall_score is the average of the five scores. These points do not change the overall architecture, but they matter for strict reproducibility (Bai et al., 16 Sep 2025).

4. Fix Generator and patch validation

Fix Generator translates root-cause understanding into concrete edits. The paper emphasizes that this is not a single-shot prompt. Instead, FVDebug uses an ensemble of prompting strategies inspired by best-of-$\mathcal{V}$2 and self-consistency. The strategies listed are full_context, suspicious_focus, narrative_focus or causal_narratives_focus, minimal_context, and bugs_and_suggestions_only. Each strategy exposes the issue differently: one presents all context, one emphasizes suspicious signals, one emphasizes causal narratives, one presents only distilled root cause, and one supplies concise bug/fix hints (Bai et al., 16 Sep 2025).

The formal outline appears as “Algorithm 3: Ensemble Fix Generation.” For each strategy, the system builds a strategy-specific prompt, parses the returned fixes, validates them against the RTL code map $\mathcal{V}$3, creates a signature for deduplication, merges duplicates across strategies, and then ranks the surviving fixes by consensus. The output JSON includes a category—RTL Bug, Under-Constraint, or Over-Constraint—an analysis, and a list of fixes. Each fix contains buggy_code, code, description, confidence, and location with module, signal, file, and line (Bai et al., 16 Sep 2025).

Validation is deliberately text-grounded. The appendix describes a three-stage validation process: exact substring matching first, then whitespace-normalized matching, then remaining fixes are discarded. CreateSignature normalizes the buggy and corrected code pair by removing whitespace variations and comments and deterministically ordering commutative operations, then hashes the result. Consensus across strategies becomes an explicit ranking signal. For a fix produced by $\mathcal{V}$4 strategies out of 5 total, the reported consensus boost is

$\mathcal{V}$5

The cap is intended to avoid over-reliance on agreement alone. The paper states that final confidence combines original generation confidence and this consensus boost, but it does not provide the exact combination formula (Bai et al., 16 Sep 2025).

A common misconception is that FVDebug is only a patch generator. The pipeline contradicts that reading. Fix generation is the last stage, and it depends on prior graph construction, suspicious-node analysis, and narrative ranking. Another common misconception is that the system only repairs RTL logic bugs. The prompts and output categories explicitly allow diagnoses such as under-constraints, over-constraints, assertion/property issues, and design-intent mismatch, although the strongest quantitative fix results are reported for RTL benchmark cases (Bai et al., 16 Sep 2025).

5. Evaluation and empirical results

The main open benchmark is SVA-Eval-Human, described as a curated collection of 38 real hardware debugging challenges with human-verified ground truth fixes. Each design includes a counterexample trace, buggy RTL, and ground-truth fix. FVDebug is also evaluated on two CVA6 RISC-V processor failures from AutoSVA and on two proprietary, production-scale FV counterexamples with masked details. For all baselines and ablations in the reported experiments, the underlying model is o3-mini (Bai et al., 16 Sep 2025).

On SVA-Eval-Human, the full system reports:

• Quality@Best = 0.956
• NDCG@5 = 0.983
• MRR = 0.858
• Kendall’s $\mathcal{V}$6
• Pass@1 = 0.711
• Pass@5 = 0.868

The baselines are markedly lower on at least some of these measures. Direct LLM reports Quality@Best 0.783, Pass@1 0.605, and Pass@5 0.658. Flat Trace Analysis reports Quality@Best 0.474, Pass@1 0.632, and Pass@5 0.816. The paper treats the gap from 0.956 to 0.474 in Quality@Best between full FVDebug and Flat Trace Analysis as its clearest evidence that explicit causal structure matters more than mere temporal formatting (Bai et al., 16 Sep 2025).

The ablations isolate the contribution of each stage. Without For-Against, the system reports Quality@Best 0.953, NDCG@5 0.969, MRR 0.817, Kendall’s $\mathcal{V}$7, Pass@1 0.632, Pass@5 0.868. Without Rover, it reports Quality@Best 0.795, NDCG@5 0.844, MRR 0.567, Kendall’s $\mathcal{V}$8, Pass@1 0.643, Pass@5 0.821. Without Ensemble, it reports Quality@Best 0.956, NDCG@5 0.983, MRR 0.858, Kendall’s $\mathcal{V}$9, Pass@1 0.684, Pass@5 0.763. These numbers support three empirical claims stated in the paper: causal structure is essential; Rover is crucial for ranking and explanation quality; and the ensemble improves robustness in fix generation, especially at Pass@5 (Bai et al., 16 Sep 2025).

The CVA6 processor cases are harder. For the MMU and LSU failures, the full system reports Quality@Best 0.713, NDCG@5 0.875, MRR 0.667, and Kendall’s $\mathcal{E}$0. Direct LLM reports 0.575, 0.801, 0.500, and 0.333. Flat Trace Analysis reports 0.475, 0.800, 0.531, and 0.100. FVDebug without Rover reports 0.300, 0.742, 0.276, and 0.294. The paper explicitly states that automated fix generation for such multi-line processor issues remains future work, so Pass@k is not reported for those cases (Bai et al., 16 Sep 2025).

The proprietary case studies are used to support industrial applicability. CEX-1 is a testbench error in which an internal token count omitted a mesh input path; FVDebug’s diagnosis was rated an “Excellent Match.” CEX-2 is a missing constraint causing FIFO overflow through failed backpressure; FVDebug identified it as acceptance of writes when full due to an unconstrained request-valid signal, rated a “Strong Match.” The reported scales are large. For CEX-1: 265 RTL files, 560,202 LOC, 123 unique signals in graph, 189 nodes, 227 edges, depth 15, 163 Jasper calls, 25 LLM calls, runtime 4m 12s. For CEX-2: 257 files, 554,983 LOC, 41 unique signals, 80 nodes, 86 edges, depth 11, 67 Jasper calls, 21 LLM calls, runtime 6m 07s (Bai et al., 16 Sep 2025).

The paper explains the metrics in words but does not provide mathematical formulas for NDCG, MRR, Kendall’s $\mathcal{E}$1, or Pass@k. Fix success is validated concretely by applying the generated fix and re-running formal verification in Jasper. That choice makes the reported Pass@1 and Pass@5 values operational rather than purely textual (Bai et al., 16 Sep 2025).

6. Position in the debugging literature and principal limitations

FVDebug belongs to a broader movement toward structured, LLM-assisted debugging, but its starting point is distinctive. “VeriDebug” learns bug location and bug type before guided Verilog correction in a shared parameter space (Wang et al., 27 Apr 2025). “VeriPilot” uses a golden reference model, internal variable alignment, and CDFG-based tracing to isolate suspicious RTL regions and their correct counterparts (Wang et al., 22 Jun 2026). “DB-Hunter” performs interactive differential testing of Vivado’s FPGA simulation debugger itself, which is a debugger-validation problem rather than a failed-property root-cause analysis problem (Guo et al., 3 Mar 2025). In that landscape, FVDebug is specifically about failed formal properties and counterexample-driven causal explanation, not generic RTL repair and not debugger correctness testing.

Several limitations are visible in the paper’s own discussion. The method depends heavily on the availability and quality of formal counterexamples and JasperGold’s visualize -why explanations. Trace depth is heuristic: the default 20-cycle cutoff is justified empirically, but the paper does not provide a systematic depth-sensitivity study. The approach also depends on specification and documentation quality, because the scanner and rover use retrieved spec excerpts to distinguish intended behavior from implemented behavior (Bai et al., 16 Sep 2025).

Scalability is managed by graph pruning and retrieval, but the lower scores on processor-scale examples show that the task remains difficult as graph size and context complexity increase. Fix generation is strong on the 38-case benchmark but not yet mature for complex multi-line processor bugs. Some implementation details remain underspecified, including the exact retriever model, the precise thresholding logic for suspiciousness, and the exact formula that combines intrinsic fix confidence with consensus. These are best read as current-system limitations rather than contradictions of the overall design (Bai et al., 16 Sep 2025).

The paper’s strongest claim is therefore not that FVDebug replaces formal tools, but that it adds an explicit causal reasoning layer between counterexample generation and engineering action. A plausible implication is that its lasting contribution lies in representation and orchestration: it turns a counterexample from a waveform artifact into a graph-structured explanation space and then stages LLM reasoning across local suspicion analysis, narrative synthesis, and validated repair.