Formal Counterexample Generation

• Formal counterexample generation is a technique that constructs precise, machine-checkable witnesses (e.g., traces, states, or critical subsystems) indicating specification violations.
• It integrates specification languages, solver technologies, and explanation interfaces to diagnose failures in contexts such as SMT-based planning, probabilistic verification, and theorem proving.
• The methodology supports iterative refinement and repair loops by using counterexamples to guide synthesis, invariant strengthening, and debugging of formal artifacts.

Formal counterexample generation is the construction of machine-checkable evidence that a formal artifact, execution, or proof obligation violates a specification. Across model checking, deductive verification, synthesis, theorem proving, and neuro-symbolic systems, the central object is not merely an informal bug report but a formally defined witness of failure: a trace, state, model, path set, critical subsystem, infeasible plan prefix, disagreement trajectory, or theorem-prover model whose correctness is established relative to the underlying semantics and oracle. The topic therefore sits at the intersection of specification languages, solver technology, and explanatory interfaces, and it has expanded from classical model checking into probabilistic verification, infinite-state systems, proof repair, neural synthesis, and large-language-model-guided reasoning (Jha et al., 2023).

1. Formal foundations and problem formulations

Formal counterexample generation is usually defined relative to a verification judgment of the form $K \vDash \varphi$, $\Psi(\mathcal{F},\pi)\models\varphi$, or $\forall \vec{x}.\,\sigma(P,\vec{x})$. When the judgment fails, the counterexample is a formal object certifying that failure. In symbolic transition systems, a counterexample is a trace $p \in \text{Traces(STS)} \setminus \varphi$; the space of all such traces can itself be organized into finitely many classes via trace constraints that cover all violating traces (Vick et al., 2021). In deductive verification, the relevant object is often a model of the negated verification condition, such as a valuation satisfying $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$, which can then be mapped back to source-level inputs and heap states (Huang et al., 2022). In second-order synthesis for loop invariants, counterexamples arise from satisfiable queries to the negation of initiation, consecution, or safety conditions, yielding either a single state $\vec{x}$ or a pair $(\vec{x},\vec{x}')$ (Polgreen et al., 2020).

A recurring distinction is between a single violating behavior and a counterexample adequate for the logic at hand. For non-probabilistic safety properties, a finite bad prefix is often sufficient. For liveness properties in finite systems, a lasso-shaped object or a finite prefix exposing a loop is standard. For probabilistic specifications such as $\mathbb{P}_{\le \lambda}(\lozenge G)$, one path is generally insufficient; instead, a counterexample is a set of paths or a critical subsystem whose accumulated probability mass exceeds the threshold (Češka et al., 2019). For continuous-state reachability, the counterexample is a closed-loop trajectory $\tau:\mathbb{R}^+\to X$ violating $(\exists t\ge 0)\,\tau(t)\in G$, together with a disagreement state where the candidate policy exits the demonstrator’s admissible control set (Ravanbakhsh et al., 2019).

This broadening of the formal object is accompanied by a broadening of the synthesis setting. In neuro-symbolic planning, the “counterexample” is an infeasible plan prefix obtained from SMT unsatisfiability analysis: the smallest prefix $\Psi(\mathcal{F},\pi)\models\varphi$0 such that the corresponding logical encoding is unsatisfiable, expressed back to the LLM as “Any plan with the following prefix is not correct” (Jha et al., 2023). In Lean-based mathematical reasoning, formal counterexample generation is explicitly posed as the task of producing both a witness and a Lean 4 proof certifying that the witness falsifies the weakened statement (Li et al., 19 Mar 2026). This suggests that the concept is best viewed not as a single data type but as a family of formally validated failure witnesses parameterized by the underlying semantics.

2. Counterexample forms and representations

The form of a counterexample depends on the verification setting, but several recurrent representations appear in the literature.

Setting	Counterexample form	Representative source
SMT-checked planning	Infeasible plan prefix	(Jha et al., 2023)
Loop invariant synthesis	State $\Psi(\mathcal{F},\pi)\models\varphi$1 or transition pair $\Psi(\mathcal{F},\pi)\models\varphi$2	(Polgreen et al., 2020)
Deductive verification	Concrete test input and violating execution	(Petiot et al., 2015)
Proof failure in Verus	Concrete program state $\Psi(\mathcal{F},\pi)\models\varphi$3 with $\Psi(\mathcal{F},\pi)\models\varphi$4	(Yang et al., 26 Mar 2026)
Probabilistic model checking	Critical subsystem or path set	(Češka et al., 2019)
CSM temporal model checking	Critical tree	(Daszczuk, 2017)

Some representations are intentionally close to the solver. In RTL-to-RTL equivalence checking for floating-point arithmetic, counterexamples are single-cycle valuations of input signals, internal stage signals such as algman, bigman, and outputs s, e, m, because the case study is combinational and failed assertions therefore yield depth-1 traces (Mohanty et al., 7 Dec 2025). In Boogie- and Z3-based verification of Eiffel, the counterexample is initially an SMT model over Heap@0, Heap@1, Select, Store, object references, and fields; Proof2Test reconstructs from this model the source-level object graph, argument values, and an executable failing test (Huang et al., 2022). In Frama-C/ACSL, the counterexample is an input for the original C code or an abstracted program variant that either violates an annotation directly or reveals a subcontract weakness (Petiot et al., 2015).

Other representations are designed for explanation rather than raw solver fidelity. The “critical tree” for QsCTL over CSM systems is a tree-shaped collection of state sequences indexed by subformula identifiers; each sequence explains why a relevant subformula has the value opposite to the desired value, and sequences “stick” so that the ending state of a parent sequence is the starting state of child sequences (Daszczuk, 2017). In robotic planning over MDPs, an explainable counterexample is a pair $\Psi(\mathcal{F},\pi)\models\varphi$5 where $\Psi(\mathcal{F},\pi)\models\varphi$6 is a critical subsystem and $\Psi(\mathcal{F},\pi)\models\varphi$7 is a finite set of structured natural-language sentences such as “The robot moves south when north of pick-up area,” required to be sound, complete, and minimal (Feng et al., 2018). In Alloy-based counterexample classification, a class is represented not by one trace but by a trace constraint $\Psi(\mathcal{F},\pi)\models\varphi$8, and a finite family of such constraints covers the full space of counterexamples (Vick et al., 2021).

The representation layer is therefore itself an active research topic. A systematic literature review reports that most studies explain counterexamples graphically or as traces, frequently minimize them, often localize errors in the model’s input format, and commonly support LTL or CTL using tools from the Symbolic Model Verifier family (Kaleeswaran et al., 2022).

3. Generation mechanisms across verification paradigms

The dominant generation mechanisms are satisfiability solving, model checking, falsification, and theorem-prover model extraction. In SMT-based planning, a candidate plan $\Psi(\mathcal{F},\pi)\models\varphi$9 is encoded as

$\forall \vec{x}.\,\sigma(P,\vec{x})$0

and Z3 is asked whether $\forall \vec{x}.\,\sigma(P,\vec{x})$1 is satisfiable. If not, the solver analysis is used to identify the smallest infeasible prefix, which becomes the counterexample fed back to the learner (Jha et al., 2023). In loop-invariant synthesis, the verifier instead asks satisfiability queries for $\forall \vec{x}.\,\sigma(P,\vec{x})$2, $\forall \vec{x}.\,\sigma(P,\vec{x})$3, and $\forall \vec{x}.\,\sigma(P,\vec{x})$4, returning models as formal counterexamples (Polgreen et al., 2020).

In probabilistic verification, the generation problem is inherently quantitative. For families of Markov chains, the verifier computes a critical set $\forall \vec{x}.\,\sigma(P,\vec{x})$5 such that the sub-chain $\forall \vec{x}.\,\sigma(P,\vec{x})$6 still violates $\forall \vec{x}.\,\sigma(P,\vec{x})$7, and then lifts this to a conflict: a partial realization $\forall \vec{x}.\,\sigma(P,\vec{x})$8 such that every full realization extending it also violates the property (Češka et al., 2019). For infinite-state chemical reaction networks, the paper develops a bounded-model-checking method using SMT to find individual witness paths of bounded length, builds a witness CTMC from the accumulated paths, and stops when the probability of reaching the target within time $\forall \vec{x}.\,\sigma(P,\vec{x})$9 in that witness CTMC exceeds the threshold $p \in \text{Traces(STS)} \setminus \varphi$0 (Ahmadi et al., 2022). The same work introduces divide-and-conquer over intermediate species-count targets and a scaffolding technique that reuses already discovered witness states to synthesize more witnesses without solving only ever larger BMC encodings.

In continuous control, counterexample generation is implemented by a falsifier rather than a satisfiability solver. The verifier optimizes over the initial state $p \in \text{Traces(STS)} \setminus \varphi$1, while controls are fixed by the candidate policy $p \in \text{Traces(STS)} \setminus \varphi$2, to maximize an MPC-style cost; trajectories that fail to reach the goal set become counterexample traces, and a disagreement state $p \in \text{Traces(STS)} \setminus \varphi$3 on such a trajectory is extracted when $p \in \text{Traces(STS)} \setminus \varphi$4 (Ravanbakhsh et al., 2019). In theorem proving and proof repair, the mechanism may again be SAT/SMT, but the challenge is source-level usability. EXVERUS addresses this by having an LLM synthesize Z3Py scripts over source variables, enumerate up to $p \in \text{Traces(STS)} \setminus \varphi$5 models, and then validate those models by replaying them in extracted Verus fragments (Yang et al., 26 Mar 2026).

A distinct lineage constructs counterexamples by transformation rather than explicit state exploration. In transformation-based model checking of reactive systems, distillation produces a simplified tail-recursive program, and a verdict function $p \in \text{Traces(STS)} \setminus \varphi$6 accumulates a state trace $p \in \text{Traces(STS)} \setminus \varphi$7 while proving or refuting LTL formulas, returning (False, π) as a counterexample or (True, π) as a witness (Hamilton, 2016). In CSM systems checked by CBS, the model checker constructs a “critical tree” by BFS-style exploration in spheres, with one sequence per formula element and rule-specific ending-state conditions for operators such as AX, AF, AG, and Uw (Daszczuk, 2017).

4. Counterexample-guided refinement and synthesis loops

A major development is the use of formal counterexamples as the primary feedback signal in iterative repair or synthesis. In classical CEGIS form, a synthesizer proposes a candidate and a verifier either returns OK or a counterexample, which is then used to refine the next candidate. This pattern is explicit in loop-invariant synthesis with neural sequence models: counterexamples are converted into new I/O examples and fed back to the synthesizer, nearly doubling the number of SyGuS invariant benchmarks solved relative to example guidance alone, from 12 to 21 (Polgreen et al., 2020). In probabilistic program sketches, critical subsystems are lifted to conflicts—partial hole assignments—and encoded as additional SMT clauses excluding all realizations extending that conflict (Češka et al., 2019).

The same idea has been adapted to LLM-centered systems. Neuro-symbolic planning uses a conversational LLM as synthesizer and Z3 as verifier; an infeasible prefix is rendered back into human-readable plan syntax and appended as a textual prohibition, after which the LLM attempts a new plan (Jha et al., 2023). EXVERUS turns verified counterexamples into a proof-repair loop: after CexGen and ValidateCex, an error triager classifies the failure as wrong_fact, too_weak, or other, and a corresponding mutator either weakens, strengthens, or otherwise repairs invariants and annotations (Yang et al., 26 Mar 2026). In floating-point RTL verification, failed SVAs produce counterexamples that localize discrepancies to Mantissa Alignment or Add-Round stages; engineers then either fix the RTL or strengthen assumptions such as exp_inputs_are_equal, repeating the process until the theorem spec.result = impl.result and its stage lemmas are proved (Mohanty et al., 7 Dec 2025).

Counterexample-guided refinement also appears outside discrete symbolic synthesis. In policy learning from demonstrations, a falsifier returns a violating trajectory, from which the learner extracts a disagreement state $p \in \text{Traces(STS)} \setminus \varphi$8 where $p \in \text{Traces(STS)} \setminus \varphi$9; this adds a new polyhedral constraint on the policy parameters $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$0, shrinking the feasible region $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$1 (Ravanbakhsh et al., 2019). Under the maximum-volume ellipsoid selection strategy and the stated compactness assumptions, the number of iterations until termination is $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$2, where $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$3 is the parameter dimension (Ravanbakhsh et al., 2019). In ACL2, cgen first supplies counterexamples and witnesses to a false theorem, and DrLA then uses theory exploration and subsumption to suggest additional hypotheses that eliminate the counterexamples while preserving at least some witnesses, such as suggesting (or (stringp x) (true-listp x)) for double reversal (Gamboa et al., 2023).

A plausible implication is that counterexamples now serve not only as certificates of falsity but as reusable training, repair, and abstraction signals. This role is most explicit in formal mathematics with Lean 4, where a symbolic mutation tactic drops a necessary hypothesis from a proven theorem, creating an existential counterexample task and a dropped-hypothesis companion task; a multi-reward expert-iteration framework then uses Lean-verified proofs of these tasks to improve both counterexample generation and theorem proving (Li et al., 19 Mar 2026).

5. Explanation, diagnosis, and human interpretability

Counterexample generation and counterexample explanation are tightly linked but not identical. Several systems explicitly transform low-level formal artifacts into representations optimized for diagnosis. In Frama-C’s STADY methodology, proof failures are classified as non-compliance, subcontract weakness, or prover incapacity by generating tests over an instrumented concrete program $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$4 and an abstracted subcontract program $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$5; the produced counterexample is therefore not only a formal witness but also a diagnostic label explaining why the proof failed (Petiot et al., 2015). Proof2Test performs a similar translation in Eiffel: from Boogie/Z3 models over heaps and fields to executable AutoTest tests that fail with the same contract violation, or, in cases of under-specified callees or loop invariants, unexpectedly pass and thereby indicate specification weakness rather than implementation error (Huang et al., 2022).

Other work explains not one counterexample but the whole space of them. Counterexample classification defines a classification $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$6 of $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$7 as a finite set of classes covering $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$8, with classes represented by trace constraints. The classification algorithm repeatedly blocks already found classes, asks the verifier for a new counterexample outside those classes, minimizes the corresponding trace constraint using minimal unsatisfiable cores, and finally removes redundant classes (Vick et al., 2021). This yields a finite summary of distinct violating behaviors such as replay attacks or man-in-the-middle attacks in Needham–Schroeder, rather than a large multiset of similar traces.

Explainability becomes even more explicit in robotic mission planning. There, the MILP variables $\neg(P \Rightarrow (b\;\mathrm{wp}\;Q))$9, $\vec{x}$0, and $\vec{x}$1 jointly optimize a critical subsystem and a minimal sentence set. The resulting explanation is required to be sound, complete, and minimal, where soundness means each selected sentence corresponds to actual behavior in the critical subsystem and completeness means every state-action choice in that subsystem is covered by exactly one sentence (Feng et al., 2018). The broader literature review confirms that graphical and trace-based explanations dominate, while probabilistic and real-time systems remain underrepresented and user studies remain rare (Kaleeswaran et al., 2022).

An objective theme across these works is that formal counterexamples are increasingly judged not only by semantic correctness but by explanatory adequacy. Shorter traces, minimal critical subsystems, structured language, source-level replay, and class-level summaries are all attempts to preserve formality while reducing cognitive load.

6. Domains, limitations, and research directions

Formal counterexample generation now spans planning, hardware equivalence, neural synthesis, continuous control, deductive verification, theorem proving, probabilistic programs, robotic planning, infinite-state CRNs, and proof repair. The domain spread is accompanied by strong differences in guarantees. Verification soundness is often explicit: if Z3 proves a plan formula satisfiable, the plan is a real solution relative to the SMT model; if $\vec{x}$2 is unsatisfiable, the infeasible prefix is a machine-checkable witness of failure (Jha et al., 2023). In neural invariant synthesis, returned invariants are sound because only SMT-verified candidates are accepted, but the synthesis procedure is incomplete and not guaranteed to terminate (Polgreen et al., 2020). In policy learning for nonlinear control, the verifier is a falsifier and therefore sound but incomplete: failing to find a violating trajectory implies only that the policy is “likely correct” (Ravanbakhsh et al., 2019).

Scalability remains a central limitation. SMT-based verification is fast for checking a single proposed solution in planning, but not used there as the search engine itself (Jha et al., 2023). In CRNs, naive BMC struggles with deep witness lengths and dense branching, motivating divide-and-conquer and scaffolding (Ahmadi et al., 2022). In floating-point RTL verification, decomposition into stage lemmas is essential because monolithic equivalence properties yield less interpretable counterexamples and more difficult proofs (Mohanty et al., 7 Dec 2025). In CSM model checking, state-space reduction preserves truth values but may destroy explanatory structure so severely that the resulting critical tree becomes nearly useless; the recommended workflow is therefore to verify on the reduced graph and reconstruct the critical tree on the original graph (Daszczuk, 2017).

LLM-integrated systems introduce additional failure modes. The planning work notes that correctness depends on the correctness of the LLM-generated Z3 encoding itself (Jha et al., 2023). EXVERUS documents both wrong informal reasoning and instruction-following failures, such as constructing an invalid function instead of formalizing a correct witness already proposed in natural language (Yang et al., 26 Mar 2026). The Lean 4 counterexample-generation work likewise notes that many synthetic mutation instances are easy or redundant, and that 7–8B models still struggle with type discipline and reuse of candidate witnesses during proof generation (Li et al., 19 Mar 2026). A plausible implication is that counterexample generation will remain a hybrid problem in which solver-side rigor and model-side semantic understanding must be co-designed, not merely concatenated.

The literature nevertheless points to clear directions. Several papers call for richer oracle interfaces than a single path or prefix, including explanation generation, fault isolation, and more informative query/response types (Jha et al., 2023). The RTL verification work shows that counterexamples can guide not only implementation debugging but also the debugging of AI-generated properties themselves (Mohanty et al., 7 Dec 2025). The systematic review highlights a need for more work on probabilistic and real-time counterexample explanation, better integration with domain-specific models, and stronger empirical evaluation with users (Kaleeswaran et al., 2022). Taken together, these results suggest that formal counterexample generation is evolving from a by-product of verification into a first-class methodology for diagnosis, synthesis, repair, and interactive reasoning.