AlgoVeri: Verified Code Generation Benchmark

• AlgoVeri is a benchmark that provides a rigorously aligned evaluation suite for vericoding, encoding 77 algorithmic problems uniformly across Dafny, Verus, and Lean 4.
• It enforces identical functional specifications and contracts across systems, enabling systematic assessment of LLMs in generating semantically correct and verifiable code.
• Empirical findings reveal a pronounced verification-paradigm gap, highlighting performance variations among auto-active verifiers like Dafny and interactive theorem provers like Lean.

AlgoVeri is a rigorously aligned benchmark for evaluating LLMs and verification-focused code generation systems on the task of “vericoding”—the automated synthesis of programs alongside machine-checked proofs of correctness. Its design addresses a longstanding gap in formal verification benchmarks by ensuring cross-system comparability: it encodes $77$ classical algorithmic problems in Dafny, Verus, and Lean 4, enforcing identical functional specifications across all languages. This alignment enables the systematic assessment of LLMs’ ability to generate semantically correct and formally verifiable code regardless of the underlying verification paradigm (Zhao et al., 10 Feb 2026).

1. Benchmark Motivation and Scope

The development of AlgoVeri was motivated by two converging trends: the increasing ability of LLMs to generate machine-verified code (vericoding) and a lack of rigorous, cross-paradigm evaluation methodologies. Existing benchmarks either restrict themselves to elementary verification tasks (e.g., simple array-bounds or arithmetic checks) or test models on idiosyncratic benchmarks in single verification ecosystems—rendering their performance metrics non-comparable.

AlgoVeri provides a unified, multilingual suite spanning a spectrum of classical algorithms, including:

• Sorting and searching routines
• Dynamic programming (e.g., Longest Increasing Subsequence)
• Advanced data structure operations (e.g., red–black tree insertion)
• Graph algorithms (e.g., Edmonds–Karp max-flow)

Each task is formalized using an identical contract—preconditions, postconditions, and auxiliary definitions—carefully encoded as ghost-state predicates in Dafny and Verus (auto-active verifiers) and as Definitions and Theorems in Lean (an interactive theorem prover). As a result, each code/proof generation system faces identical logical and functional challenges independent of language or toolchain (Zhao et al., 10 Feb 2026).

2. Task Design and Specification Formalism

Tasks in AlgoVeri are specified by fully aligned contracts. For each problem $P$, the precondition and postcondition expressed in first-order logic are exactly mirrored in Dafny, Verus, and Lean 4. Canonical examples include:

• Longest Increasing Subsequence
  • Precondition: $$\forall k < m \;.\; 0 \leq k < m < |s| \Rightarrow s[k] < s[m]$$
  • Postcondition: $$\exists is:\text{seq}\langle\text{int}\rangle\ .\ \mathrm{is\_valid\_is}(s,is) \wedge |is| = result \wedge \forall sub\ .\ \mathrm{is\_valid\_is}(s,sub) \Rightarrow |sub| \leq result$$
• Max-Flow (Edmonds–Karp)
  • Precondition: $$\mathrm{well\_formed}(G) \wedge \mathrm{capacities\_bounded}(G) \wedge s \neq t$$
  • Postcondition: $$\exists f\ .\ \mathrm{is\_max\_flow}(G,f,s,t) \wedge \mathrm{flow\_val}(G,f,s,t) = max\_val$$

These contracts are encoded as SMT-friendly predicates and verification logic that require solution code to not only implement the algorithm but also substantiate its properties via formal proof artifacts (loop invariants, ghost code, or tactic scripts as appropriate to the language) (Zhao et al., 10 Feb 2026).

3. Evaluation Protocol and Model Assessment

Benchmarking on AlgoVeri proceeds in a two-stage, multi-turn protocol:

1. Code/Proof Generation: The LLM receives the natural-language problem description with code stubs (function signature, spec, and empty body) in one system and is tasked with producing both the implementation and the proof artifacts such that the verifier (Dafny, Verus, or Lean) accepts the submission.
2. Iterative Repair: Up to $15$ rounds of “compile-and-repair” are allowed. The model observes verifier error messages and attempts to refine its submission, simulating realistic developer-debug feedback cycles.
3. Semantic Filtering: To ensure specification compliance is meaningful (preventing “spec-gaming,” where trivial code satisfies formal contracts but fails to implement the intended algorithm), candidate solutions are checked by an LLM-powered semantic checker that examines whether the correct algorithm (not just the correct contract) was synthesized (Zhao et al., 10 Feb 2026).

4. Empirical Findings: Paradigm Gap and Performance Analysis

Experiments on AlgoVeri with Gemini-3 Flash (proprietary) and GPT-OSS-120B (open-source) highlight a pronounced “verification-paradigm gap”:

System	Compiler-Verified (%)	Semantic-Pass (%) (after filtering)
Gemini-3 (Dafny)	55.8	40.3
Gemini-3 (Verus)	25.97	24.7
Gemini-3 (Lean)	9.09	7.8
GPT-OSS (Dafny)	21.0	lower (exact filtered rate not provided)
GPT-OSS (Verus)	7.7	lower
GPT-OSS (Lean)	12.6	lower

These results demonstrate:

• LLMs can achieve tractable verification rates in Dafny, owing to high-level abstractions, unbounded mathematical types, and powerful SMT automation.
• Performance degrades on Verus due to explicit systems-level memory tracking, ownership annotations, and low-level Rust-macro syntax, which impose syntactic and type-theoretic barriers.
• In Lean, performance collapses further, as explicit proof scripting and combinatorial lemma retrieval pose a search problem of substantially higher complexity (Zhao et al., 10 Feb 2026).

5. Computation Dynamics: Iterative Repair and Model Feedback

Exploration of test-time feedback usage reveals divergent compute utilization:

• Gemini-3 Flash: Pass rates increase substantially with more repair iterations, nearly tripling in Dafny and doubling in Verus from initialization to Round 15. This demonstrates the model’s capacity to learn from error feedback and progressively correct its output.
• GPT-OSS-120B: Pass rates saturate by the fourth round of repair, with little to no improvement from further iterations. An iso-compute “depth vs. width” analysis establishes that GPT-OSS benefits more from brute-force sampling than from deep iterative repair, indicating limited ability to translate compiler feedback into substantive corrections (Zhao et al., 10 Feb 2026).

6. Error Analysis and Refinement Trajectory

Analysis of model failures and refinement patterns reveals error-mode stratification tied to verification paradigms:

• Dafny: The majority of initial failures are syntax or type mismatches, which models typically overcome rapidly, leaving only pure SMT proof failures for later rounds.
• Verus: Syntax and type errors—especially those related to ownership and casting—persist across all rounds, forming an enduring “macro barrier” that stymies logical progress.
• Lean: After resolving surface syntax, error modes bifurcate into hallucinations (invention of non-existent tactics or lemmas) and intrinsically hard verification failures (unproven obligations even when tactics are syntactically correct).

This paradigm-specific error structure highlights that Dafny is primarily logic-focused, Verus is syntax-focused, and Lean is proof search–focused (Zhao et al., 10 Feb 2026).

7. Implications for Verification Research and Future Directions

Findings from AlgoVeri point toward specific directions for future formal verification and vericoding benchmarks:

• Paradigm Alignment: True cross-system comparability requires perfectly aligned task suites, as idiosyncratic contracts or language features can artificially inflate (or deflate) model performance.
• Syntactic Extraction and Type System Coverage: Improving model capacity to handle systems-level abstractions (memory, ownership, casting) and rich type systems, as exemplified by persistent challenges in Verus.
• Search-Augmented Theorem Proving: Surmounting the complexity of proof search in ITPs such as Lean likely necessitates hybrid approaches combining learned proof synthesis with retrieval and repair based on existing libraries.
• Benchmark Utility: AlgoVeri establishes a rigorous litmus test for next-generation LLMs in verified code generation, especially in synthesizing global invariants, ghost code, and existential-optimality proofs involving nontrivial algorithm classes.

This suggests that while high-level SMT-based auto-active verifiers are already accessible to leading LLMs, realistic systems-level and ITP domains remain at the research frontier—necessitating advances both in language modeling and in the modeling of proof state search dynamics (Zhao et al., 10 Feb 2026).