LiveCodeBench-Pro-Dafny Benchmark
- LiveCodeBench-Pro-Dafny is a comprehensive benchmark for verified program synthesis that challenges models to generate both executable Dafny code and the necessary formal proof artifacts.
- It employs a verifier-based evaluation, measuring success through verifier acceptance and executable test outcomes to highlight distinctions between functional correctness and runtime performance.
- The benchmark consists of 250 contest-style problems split into easy, medium, and hard levels, with rigorous specification auditing to ensure accurate formal verification.
LiveCodeBench-Pro-Dafny (LCB-Pro-Dafny) denotes a Dafny benchmark for verified code generation in which a model must produce executable Dafny code together with the proof artifacts required for verification. In "AxDafny: Agentic Verified Code Generation in Dafny" (Breen et al., 30 Jun 2026), it is introduced as a benchmark of 250 competition-style programming problems translated into Dafny with formal specifications and a verifier-based evaluation harness. In the broader vericoding literature, the same label is also used for the Dafny component of a larger benchmark for "vericoding," that is, LLM generation of formally verified code from formal specifications (Bursuc et al., 26 Sep 2025). Both usages place Dafny in a setting where success is defined by verifier acceptance rather than by plausible-looking code alone.
1. Definition and problem setting
LCB-Pro-Dafny is designed to study end-to-end verified program synthesis rather than only proof-hint completion. The benchmark requires the generation of both executable code and the verification artifacts needed for Dafny to accept the program (Breen et al., 30 Jun 2026). This distinguishes it from earlier Dafny tasks in which the implementation already exists and the model only restores missing invariants or assertions.
The underlying programming and verification substrate is Dafny, a language and verifier for static functional correctness. Dafny supports requires for preconditions, ensures for postconditions, loop invariants, local assert statements, and termination arguments via decreases; it verifies that code satisfies those annotations for all executions (Gauci, 2014). Within LCB-Pro-Dafny, a task is therefore not solved by producing output-equivalent code alone, but by producing code whose behavior is formally established against a fixed specification.
The benchmark is also situated within the distinction between vericoding and vibe coding. The vericoding formulation starts from a formal specification and asks the model to produce implementation and proof so that the result is machine-checked correct, whereas vibe coding starts from a natural-language description and asks the model to write potentially buggy code (Bursuc et al., 26 Sep 2025). LCB-Pro-Dafny adopts the former orientation, even when natural-language problem statements are present as part of the task.
2. Benchmark lineage and relation to prior Dafny evaluations
The benchmark is positioned against several earlier Dafny benchmarks with different task definitions. DafnyBench gives the model an implementation and specification and asks it to restore missing invariants and assertions; the AxDafny paper characterizes this as a proof-hint synthesis task (Breen et al., 30 Jun 2026). DafnyBench itself contains 782 stand-alone Dafny programs that compile, constructed from a GitHub scrape, Clover, and dafny-synthesis, and evaluates whether a model can reconstruct removed assert and invariant statements while preserving requires and ensures and avoiding {:verify false} or assume false (Loughridge et al., 2024).
By contrast, LCB-Pro-Dafny is built from LiveCodeBench-Pro problems and is explicitly intended to resemble competition-style programming problems rather than only proof-centric repair tasks (Breen et al., 30 Jun 2026). The paper also contrasts it with DafnyComp / VeriEquivBench, which focus on specification and proof quality rather than end-to-end generation, and with dafny-synthesis and HumanEval-Dafny, which are described as smaller and less competition-programming-oriented.
A separate nearby benchmark family adds an additional source of terminological confusion. "A benchmark for vericoding: formally verified program synthesis" uses LCB-Pro-Dafny for the Dafny portion of a multilingual vericoding benchmark containing 12,504 formal specifications overall, including 3,029 Dafny tasks (Bursuc et al., 26 Sep 2025). This suggests that the name has become attached both to a 250-problem competition-style Dafny benchmark and to a broader Dafny slice inside a larger vericoding suite.
3. Dataset construction and curation
In the AxDafny formulation, LCB-Pro-Dafny contains 250 LiveCodeBench-Pro problems with the following split (Breen et al., 30 Jun 2026):
| Split | Problems |
|---|---|
| Easy | 100 |
| Medium | 100 |
| Hard | 50 |
The benchmark retained the original difficulty labels from LiveCodeBench-Pro. Each instance was converted into a Dafny task containing the original natural-language problem statement, a Dafny method signature, and a formal specification. The specifications were initially produced with model assistance, then manually reviewed. The curation process removed or revised tasks with inconsistent specifications, underspecified specifications, and vacuous specifications, and the authors state that they checked that the final specs captured the intended functional behavior of the original problem (Breen et al., 30 Jun 2026).
The appendix-level curation details are unusually explicit. All 250 released specs parse and type-resolve under Dafny 4.11.0. Under dafny resolve --allow-warnings, there were no missing ensures clauses and no trivially true postconditions. Under stricter default warnings, 49 specs emitted quantifier-trigger warnings but still parsed and type-resolved. An LLM-as-judge review flagged 14/250 specifications (5.6%) for semantic mismatches, and all were fixed before inclusion. In one representative repair, a set of chosen objects had been modeled as a sequence without forbidding duplicates; this was corrected by requiring the sequence to be distinct. As an additional vacuity check, no placeholder implementation verified, and no automatically generated constant-or-empty-output mutation verified in the 234 cases where the mutation harness applied (Breen et al., 30 Jun 2026).
These curation steps matter because verified-code-generation benchmarks can otherwise be distorted by weak or semantically off-target specifications. A plausible implication is that LCB-Pro-Dafny was designed not merely as a translation exercise, but as a specification-audited benchmark in which verifier success is intended to track the original programming problem as closely as possible.
4. Task structure and verifier-based evaluation
Each task presents a Dafny file in which the model must produce the implementation and proof structure while leaving the provided specification intact. The benchmark expects the use of standard Dafny proof structure: requires for preconditions, ensures for postconditions, loop invariants, local assert statements, and termination arguments (Breen et al., 30 Jun 2026).
The evaluation harness accepts a submission only if all of the following hold: the generated Dafny implementation verifies; the provided specification, definitions, and functions are not modified; and the solution does not use vacuous proof shortcuts, such as assume, {:extern}, or {:verify false} (Breen et al., 30 Jun 2026). The harness also checks for proof-bypass behavior and rejects cheating patterns.
The primary metric is verifier acceptance. A solution counts as correct only if it verifies in Dafny, preserves the fixed specification, and avoids vacuous proof shortcuts. The benchmark also reports a secondary executable metric: verified Dafny solutions are compiled to Python and run against the original LiveCodeBench-Pro test harness under the original time limits (Breen et al., 30 Jun 2026). The paper is explicit that this secondary metric is not the same as verification.
This evaluation logic is consistent with earlier Dafny verification benchmarks. DafnyBench likewise counts a solution as successful only if the reconstructed program is verified by Dafny, preserves requires and ensures, and does not use {:verify false} or assume false (Loughridge et al., 2024). The difference is that LCB-Pro-Dafny extends this logic from annotation repair to full program-and-proof synthesis.
5. Baselines, AxDafny results, and observed failure modes
The main LCB-Pro-Dafny baseline is direct GPT-5.5 pass@1 generation. On this benchmark, the paper reports the following verification results (Breen et al., 30 Jun 2026):
| System | Overall | Easy | Medium | Hard |
|---|---|---|---|---|
| GPT-5.5 pass@1 | 11.6% | 13.0% | 14.0% | 4.0% |
| AxDafny | 56.4% | 75.0% | 52.0% | 28.0% |
AxDafny therefore improves overall verification success by 44.8 percentage points over the GPT-5.5 pass@1 baseline. The paper also states that performance is still far from saturated on the hard split, treating that as evidence that the benchmark remains challenging for verified program synthesis (Breen et al., 30 Jun 2026).
AxDafny’s broader system is an iterative generate–check–repair framework with a proposer that writes Dafny code, a reviewer system that checks for spec weakening, proof-bypass constructs, and verifier failures, and a memory module that preserves lessons across iterations. The reviewer feeds back Dafny diagnostics, counterexamples when available, and rejected constructs or spec mismatches. Experiments allow up to 20 iterative attempts per benchmark instance (Breen et al., 30 Jun 2026).
A central empirical result is that verification success and runtime test success are different. Among AxDafny’s verified outputs on the easy split, it verifies 75/100 tasks, but among those verified solutions only 32 pass executable tests, while 39 fail by TLE and 4 fail by MLE. On the medium split, it verifies 52/100 tasks, but among those verified solutions only 6 pass executable tests, while 44 fail by TLE and 2 fail by MLE (Breen et al., 30 Jun 2026). The authors attribute this divergence to the fact that Dafny specs focus on functional correctness, not asymptotic complexity or memory usage; they also note that Dafny-to-Python compilation may introduce overhead for constructs such as sets and maps, although they partially mitigate this with a lightweight Python compatibility layer.
The AxDafny paper also reports transfer to a separate benchmark: on DafnyBench, AxDafny with Gemini-3.1-Pro verifies 725/782 = 92.7%, outperforming the strongest prior proof-hint baseline by 6.5 percentage points (Breen et al., 30 Jun 2026). In DafnyBench itself, the best previously reported result was about 68% with Claude 3 Opus under up to 10 attempts and verifier-feedback retries (Loughridge et al., 2024). This places LCB-Pro-Dafny in a broader trajectory of rapidly improving LLM performance on Dafny verification tasks.
6. Terminological ambiguity, neighboring benchmarks, and interpretation
The label LCB-Pro-Dafny should be interpreted with care. In the AxDafny paper, it denotes the 250-problem competition-style benchmark just described (Breen et al., 30 Jun 2026). In "A benchmark for vericoding: formally verified program synthesis," however, the label refers to the Dafny component of a larger multilingual benchmark containing 3,029 Dafny specs out of 12,504 formal specifications overall, with 6,174 new unseen problems (Bursuc et al., 26 Sep 2025). That broader benchmark reports a model union Dafny success rate of 82.2%, while also noting a separate improvement on original DafnyBench verification from 68% to 96% for the model union over roughly a year (Bursuc et al., 26 Sep 2025). The two uses are compatible in theme but not identical in benchmark definition, size, or task construction.
A second common confusion concerns Multi-LCB. Although it extends LiveCodeBench to multiple programming languages, its supported set is explicitly C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin, and Scala, and Dafny does not appear anywhere in the benchmark language list, the tables, the runtime/compiler environment section, or the evaluation results (Ivanova et al., 18 Jun 2026). Multi-LCB is therefore not an implementation of LCB-Pro-Dafny, though its contamination-aware design and format-conversion methodology are relevant as a blueprint for possible future extensions.
The broader vericoding literature also sharpens how LCB-Pro-Dafny should be understood conceptually. The multilingual vericoding benchmark argues that formally verified synthesis is a more rigorous alternative to natural-language-only code generation because the goal is not merely plausible code but code whose correctness is formally established against a human-written specification (Bursuc et al., 26 Sep 2025). In a separate experiment on Verina, that paper found that adding informal descriptions provided no statistically significant performance improvement and was slightly worse on average. This suggests that, for benchmarks such as LCB-Pro-Dafny, the decisive artifact is the formal specification rather than auxiliary natural-language description.
Taken together, these benchmark families indicate a shift in Dafny evaluation from proof-hint reconstruction toward verified program synthesis, and from verifier-only success toward a more differentiated view in which specification fidelity, anti-cheating validation, and executable behavior under resource limits are treated as distinct dimensions of performance.