Verina: Verifiable Code Generation Arena

• Verina is a benchmark for assessing LLMs in generating Lean 4 code, formal specifications, and machine-checked proofs through a modular evaluation framework.
• It decomposes the process into three tasks—CodeGen, SpecGen, and ProofGen—with each task rigorously evaluated against detailed test suites and adversarial extensions.
• Empirical studies using Verina reveal gaps in proof generation and specification synthesis, motivating advances in traceable refinement and recursive inference techniques.

Verina, the Verifiable Code Generation Arena, is a comprehensive benchmark for evaluating LLMs in their ability to generate not only code, but also formal specifications and machine-checked proofs of correctness in the Lean 4 proof assistant. Designed to fill gaps left by prior code and specification benchmarks, Verina enables granular assessment of the strengths and limitations of LLMs in verifiable code generation—a paradigm aspiring to scale formal software guarantees via automated code, spec, and proof synthesis in concert (Ye et al., 29 May 2025).

1. Motivation and Benchmark Design

Verina addresses the critical need for rigorous, end-to-end evaluation of verifiable code generation workflows. As LLM-generated code becomes widespread in software engineering, issues of correctness and latent defects or vulnerabilities—exacerbated by the probabilistic nature of LLM outputs—are traditionally mitigated only through manual review. Formal verification, while capable of providing correctness guarantees, historically requires expert-written specifications and proofs, limiting its practical deployment.

To bridge this gap, Verina focuses on the emerging paradigm of verifiable code generation, in which an LLM produces:

• Code implementing a developer’s intent,
• A formal specification (pre- and postconditions) of its behavior,
• A proof that the code satisfies the specification.

Verina is structured as a rigorous, modular benchmark for these three subtasks (CodeGen, SpecGen, ProofGen), allowing both isolated and pipeline evaluation. The full suite consists of 189 Lean-based tasks with detailed descriptions, reference implementations, formal specs, and comprehensive test suites. Tasks were curated from MBPP‐DFY‐50 (translated from Dafny), CloverBench, and advanced student submissions from Lean verification courses, covering both basic (108 tasks) and advanced (81 tasks) levels (Ye et al., 29 May 2025).

Each Verina task comprises a Lean file with:

1. Natural-language task description,
2. Reference code (Lean 4 function),
3. Ground-truth pre- and postcondition specification,
4. Optional ground-truth proof (46/189),
5. 100% line-coverage test suite (median: 5 positive, 12 negative cases).

2. Formal Evaluation Framework

The Verina benchmark formalizes evaluation using a precise semantic framework. Let $\varphi$ be the set of all correct programs (implementing intent) and $\widehat{\varphi}$ the set characterized by a generated specification. Soundness requires $\widehat{\varphi} \subseteq \varphi$, while completeness is $\varphi \subseteq \widehat{\varphi}$.

Concrete soundness and completeness conditions for generated specs are:

• Soundness: $\forall x.\ \widehat{P}(x) \implies P(x)$ (pre), $$\forall x, y.\ \widehat{P}(x) \wedge \widehat{Q}(x, y) \implies Q(x, y)$$ (post);
• Completeness: $\forall x.\ P(x) \implies \widehat{P}(x)$ (pre), $$\forall x, y.\ P(x) \wedge Q(x, y) \implies \widehat{Q}(x, y)$$ (post).

Proof success is defined by successful Lean 4 compilation with no use of the sorry placeholder.

Task evaluation is as follows:

• CodeGen: Given a problem description (and optionally spec), the task is to generate Lean code. The metric is pass@k on the positive test suite (all must succeed).
• SpecGen: Given description (and optionally code), generate correct pre/post-conditions. A testing-based evaluator determines pass@k for soundness and completeness over the test suite.
• ProofGen: Given description, signature, code, and spec, generate a correct Lean proof (proof passes if Lean accepts it, with iterative proof refinement permitted up to 64 rounds).

3. Composition, Variants, and Extensions

Verina supports compositional evaluation by chaining subtasks in realistic usage pipelines:

• Spec-guided code generation,
• Fully end-to-end generation from description to code, spec, and proof.

Two major adversarial extensions, VerinaPlus and VerinaLite, derived via the VeriScale framework (Bai et al., 21 May 2026), address limitations of small, undemanding test suites. VeriScale performs:

• Test-suite expansion: Generating diverse, type-aware, adversarial inputs and outputs, exploiting weaknesses in specs and implementations. For VerinaPlus, test-suite size grows by over 83$\times$ (averages: 370 positive, 1114 negative-output, 119 negative-input cases per task).
• Reduction: Compacting large suites into sets (VerinaLite, $\sim$14$\widehat{\varphi}$0 expansion) with preserved discriminative power via boundary-preserving and adversary-killing heuristics.

Empirical evaluation with eight state-of-the-art LLMs shows that the expanded VerinaPlus dramatically lowers pass rates compared to the original Verina (e.g., GPT-5.5 SpecGen drops from 68.78% to 44.44%), systematically uncovering weaknesses otherwise hidden by sparse test suites. VerinaLite, nearly as discriminative, requires only $\widehat{\varphi}$11.4$\widehat{\varphi}$2 the evaluation cost (Bai et al., 21 May 2026).

4. Research Insights and Model Performance

Verina’s initial studies found that while LLMs demonstrate capacity in generating Lean code (OpenAI o4-mini: 61.4%), spec synthesis lags (51.0%), and proof generation remains a fundamental bottleneck (≤3.6% pass@1 across models). Advanced tasks are $\widehat{\varphi}$3–$\widehat{\varphi}$4 harder than basics, and iterative proof refinement can improve success on basics (e.g., o4-mini: $\widehat{\varphi}$5 at $\widehat{\varphi}$6), but only marginally on advanced tasks (Ye et al., 29 May 2025).

The gap between SpecGen and CodeGen is accentuated under adversarial testing. On VerinaPlus, SpecGen scores drop 24 points for top LLMs, surfacing the challenge of generating precise, sound, and complete specs. The best WybeCoder configuration solves up to 74% of Verina tasks, a three-fold improvement over DS Prover V2, attesting to gains from agentic verification, but proof synthesis, multi-invariant discovery, and interactive repair remain limiting (Gloeckle et al., 31 Mar 2026).

Common failure modes include:

• CodeGen: Hallucinatory library calls, off-by-one logic errors;
• SpecGen: Overly strict or too permissive specs, inability to handle complex quantifiers;
• ProofGen: Brittleness in proof strategies, missing helper lemmas, slow convergence under error message guidance.

5. Significant Methodological Advances and Future Directions

Research building on Verina has introduced methods that significantly improve specification and verification metrics:

• Traceable refinement (VeriSpecGen): Decomposition of natural language descriptions into atomic requirements, requirement-targeted test generation, and clause-level repair. This approach, validated on 100 Verina tasks, raises Claude Opus 4.5’s SpecGen pass@1 from 59.0% (2-shot baseline) to 86.6%; similar relative gains are seen for other models (Ye et al., 12 Apr 2026). Key ablations confirm the necessity of explicit requirement decomposition, grounded testing, traceable refinement, and adversarial branching.
• Recursive inference and proof revisers: Whole-task decomposition and proof revision scaffolds demonstrate that hierarchical search with iterative repair can solve formerly unattainable tasks (7/42 in Verina$\widehat{\varphi}$7), where direct generation fails completely (Tan, 29 May 2026).

Notable recommendations for the evolution of the Verina suite and its successors include integrating SMT tactics, LLM-driven provers, data scaling via LLM-assisted annotation, joint training for compositional objectives, and improved methods for avoiding trivial definitional equivalence in generated artifacts (Ye et al., 29 May 2025, Gloeckle et al., 31 Mar 2026).

6. Structural Properties and Representative Problem Types

Verina targets a spectrum of algorithmic and data-structure problems, spanning arithmetic routines (e.g., $\widehat{\varphi}$8 exponentiation), list scans, search/find queries, combinatorics, dynamic programming (Kadane’s maximum subarray), and operations on data structures. Tasks often require matching a Lean function $\widehat{\varphi}$9 to a precondition $\widehat{\varphi} \subseteq \varphi$0, postcondition $\widehat{\varphi} \subseteq \varphi$1, and potentially a fully formal proof. Several representative signatures include:

• Power 4: Ensures $\widehat{\varphi} \subseteq \varphi$2.
• CountLessThan: Ensures $\widehat{\varphi} \subseteq \varphi$3.
• SmallestMissingNumber: Requires input-sortedness and ensures $\widehat{\varphi} \subseteq \varphi$4 and $\widehat{\varphi} \subseteq \varphi$5 (Gloeckle et al., 31 Mar 2026).

A major insight is that certain functional specs can enable “reflexivity” proofs via the direct use of the specification in the return value—future benchmarks are advised to enforce a separation between implementation and specification to prevent such leakage (Gloeckle et al., 31 Mar 2026).

7. Impact, Limitations, and Directions for Benchmark Evolution

Verina’s contribution is a rigorous, high-fidelity, and modular evaluation resource for research in verifiable code generation. Its design has catalyzed new methods for traceable refinement, adversarial evaluation, and agentic verification workflows, and paved the way for benchmarks involving imperative verification through mechanical translation protocols (e.g., the Velvet framework, WybeCoder) (Gloeckle et al., 31 Mar 2026).

However, key limitations remain:

• ProofGen success rates are low, especially for complex advanced tasks.
• Some specs are susceptible to trivial solution leakage.
• Evaluation on small, non-adversarial test suites overestimates model capabilities.

Future directions include:

• Data scaling via semi-automated curation,
• Integration of SMT/ATP techniques in proof/evaluator loops,
• Enhanced techniques for defining and detecting nontrivial generative artifacts,
• Broader benchmark mixing between functional and imperative verification modalities,
• Development of fine-grained evaluation tasks that stress invariant discovery and interactive proof repair (Ye et al., 29 May 2025, Bai et al., 21 May 2026, Ye et al., 12 Apr 2026, Gloeckle et al., 31 Mar 2026, Tan, 29 May 2026).

Verina thus remains a central resource for both methodology development and empirical evaluation in the automated, LLM-driven formal verification community.