---
title: LiveCodeBench-Pro-Dafny Benchmark
url: https://www.emergentmind.com/topics/livecodebench-pro-dafny-lcb-pro-dafny
type: topic
---

# LiveCodeBench-Pro-Dafny Benchmark

LiveCodeBench-Pro-Dafny (LCB-Pro-Dafny) denotes a Dafny benchmark for verified code generation in which a model must produce executable Dafny code together with the proof artifacts required for verification. In "AxDafny: Agentic Verified Code Generation in Dafny" [2606.32007], it is introduced as a benchmark of 250 competition-style programming problems translated into Dafny with formal specifications and a verifier-based evaluation harness. In the broader vericoding literature, the same label is also used for the Dafny component of a larger benchmark for "vericoding," that is, LLM generation of formally verified code from formal specifications [2509.22908]. Both usages place Dafny in a setting where success is defined by verifier acceptance rather than by plausible-looking code alone.

## 1. Definition and problem setting

LCB-Pro-Dafny is designed to study **end-to-end verified program synthesis** rather than only proof-hint completion. The benchmark requires the generation of **both executable code and the verification artifacts** needed for Dafny to accept the program [2606.32007]. This distinguishes it from earlier Dafny tasks in which the implementation already exists and the model only restores missing invariants or assertions.

The underlying programming and verification substrate is Dafny, a language and verifier for **static functional correctness**. Dafny supports `requires` for preconditions, `ensures` for postconditions, loop `invariant`s, local `assert` statements, and termination arguments via `decreases`; it verifies that code satisfies those annotations for all executions [1412.4395]. Within LCB-Pro-Dafny, a task is therefore not solved by producing output-equivalent code alone, but by producing code whose behavior is formally established against a fixed specification.

The benchmark is also situated within the distinction between **vericoding** and **vibe coding**. The vericoding formulation starts from a formal specification and asks the model to produce implementation and proof so that the result is machine-checked correct, whereas vibe coding starts from a natural-language description and asks the model to write potentially buggy code [2509.22908]. LCB-Pro-Dafny adopts the former orientation, even when natural-language problem statements are present as part of the task.

## 2. Benchmark lineage and relation to prior Dafny evaluations

The benchmark is positioned against several earlier Dafny benchmarks with different task definitions. **DafnyBench** gives the model an implementation and specification and asks it to restore missing invariants and assertions; the AxDafny paper characterizes this as a **proof-hint synthesis** task [2606.32007]. DafnyBench itself contains **782 stand-alone Dafny programs that compile**, constructed from a GitHub scrape, Clover, and dafny-synthesis, and evaluates whether a model can reconstruct removed `assert` and `invariant` statements while preserving `requires` and `ensures` and avoiding `{:verify false}` or `assume false` [2406.08467].

By contrast, LCB-Pro-Dafny is built from **LiveCodeBench-Pro** problems and is explicitly intended to resemble **competition-style programming problems** rather than only proof-centric repair tasks [2606.32007]. The paper also contrasts it with **DafnyComp / VeriEquivBench**, which focus on specification and proof quality rather than end-to-end generation, and with **dafny-synthesis** and **HumanEval-Dafny**, which are described as smaller and less competition-programming-oriented.

A separate nearby benchmark family adds an additional source of terminological confusion. "A benchmark for vericoding: formally verified program synthesis" uses **LCB-Pro-Dafny** for the Dafny portion of a multilingual vericoding benchmark containing **12,504 formal specifications** overall, including **3,029 Dafny** tasks [2509.22908]. This suggests that the name has become attached both to a 250-problem competition-style Dafny benchmark and to a broader Dafny slice inside a larger vericoding suite.

## 3. Dataset construction and curation

In the AxDafny formulation, LCB-Pro-Dafny contains **250 LiveCodeBench-Pro problems** with the following split [2606.32007]:

| Split | Problems |
|---|---:|
| Easy | 100 |
| Medium | 100 |
| Hard | 50 |

The benchmark **retained the original difficulty labels** from LiveCodeBench-Pro. Each instance was converted into a Dafny task containing the **original natural-language problem statement**, a **Dafny method signature**, and a **formal specification**. The specifications were initially produced with **model assistance**, then **manually reviewed**. The curation process removed or revised tasks with **inconsistent specifications**, **underspecified specifications**, and **vacuous specifications**, and the authors state that they checked that the final specs captured the intended functional behavior of the original problem [2606.32007].

The appendix-level curation details are unusually explicit. All **250 released specs parse and type-resolve under Dafny 4.11.0**. Under `dafny resolve --allow-warnings`, there were **no missing ensures clauses** and **no trivially true postconditions**. Under stricter default warnings, **49 specs** emitted quantifier-trigger warnings but still parsed and type-resolved. An **LLM-as-judge review flagged 14/250 specifications (5.6%)** for semantic mismatches, and all were fixed before inclusion. In one representative repair, a set of chosen objects had been modeled as a sequence without forbidding duplicates; this was corrected by requiring the sequence to be **distinct**. As an additional vacuity check, **no placeholder implementation verified**, and no automatically generated constant-or-empty-output mutation verified in the **234** cases where the mutation harness applied [2606.32007].

These curation steps matter because verified-code-generation benchmarks can otherwise be distorted by weak or semantically off-target specifications. A plausible implication is that LCB-Pro-Dafny was designed not merely as a translation exercise, but as a specification-audited benchmark in which verifier success is intended to track the original programming problem as closely as possible.

## 4. Task structure and verifier-based evaluation

Each task presents a Dafny file in which the model must produce the implementation and proof structure while leaving the provided specification intact. The benchmark expects the use of standard Dafny proof structure: `requires` for preconditions, `ensures` for postconditions, loop invariants, local `assert` statements, and termination arguments [2606.32007].

The evaluation harness accepts a submission only if all of the following hold: the generated Dafny implementation **verifies**; the provided **specification, definitions, and functions are not modified**; and the solution does **not use vacuous proof shortcuts**, such as `assume`, `{:extern}`, or `{:verify false}` [2606.32007]. The harness also checks for proof-bypass behavior and rejects cheating patterns.

The primary metric is **verifier acceptance**. A solution counts as correct only if it verifies in Dafny, preserves the fixed specification, and avoids vacuous proof shortcuts. The benchmark also reports a secondary executable metric: verified Dafny solutions are compiled to Python and run against the original **LiveCodeBench-Pro test harness** under the original time limits [2606.32007]. The paper is explicit that this secondary metric is **not the same as verification**.

This evaluation logic is consistent with earlier Dafny verification benchmarks. DafnyBench likewise counts a solution as successful only if the reconstructed program is verified by Dafny, preserves `requires` and `ensures`, and does not use `{:verify false}` or `assume false` [2406.08467]. The difference is that LCB-Pro-Dafny extends this logic from annotation repair to full program-and-proof synthesis.

## 5. Baselines, AxDafny results, and observed failure modes

The main LCB-Pro-Dafny baseline is **direct GPT-5.5 pass@1 generation**. On this benchmark, the paper reports the following verification results [2606.32007]:

| System | Overall | Easy | Medium | Hard |
|---|---:|---:|---:|---:|
| GPT-5.5 pass@1 | 11.6% | 13.0% | 14.0% | 4.0% |
| AxDafny | 56.4% | 75.0% | 52.0% | 28.0% |

AxDafny therefore improves overall verification success by **44.8 percentage points** over the GPT-5.5 pass@1 baseline. The paper also states that performance is still far from saturated on the hard split, treating that as evidence that the benchmark remains challenging for verified program synthesis [2606.32007].

AxDafny’s broader system is an iterative **generate–check–repair** framework with a **proposer** that writes Dafny code, a **reviewer system** that checks for spec weakening, proof-bypass constructs, and verifier failures, and a **memory module** that preserves lessons across iterations. The reviewer feeds back Dafny diagnostics, counterexamples when available, and rejected constructs or spec mismatches. Experiments allow up to **20 iterative attempts per benchmark instance** [2606.32007].

A central empirical result is that **verification success and runtime test success are different**. Among AxDafny’s verified outputs on the **easy** split, it verifies **75/100** tasks, but among those verified solutions only **32** pass executable tests, while **39** fail by **TLE** and **4** fail by **MLE**. On the **medium** split, it verifies **52/100** tasks, but among those verified solutions only **6** pass executable tests, while **44** fail by **TLE** and **2** fail by **MLE** [2606.32007]. The authors attribute this divergence to the fact that Dafny specs focus on **functional correctness**, not asymptotic complexity or memory usage; they also note that Dafny-to-Python compilation may introduce overhead for constructs such as sets and maps, although they partially mitigate this with a lightweight Python compatibility layer.

The AxDafny paper also reports transfer to a separate benchmark: on **DafnyBench**, AxDafny with **Gemini-3.1-Pro** verifies **725/782 = 92.7%**, outperforming the strongest prior proof-hint baseline by **6.5 percentage points** [2606.32007]. In DafnyBench itself, the best previously reported result was about **68%** with Claude 3 Opus under up to **10 attempts** and verifier-feedback retries [2406.08467]. This places LCB-Pro-Dafny in a broader trajectory of rapidly improving LLM performance on Dafny verification tasks.

## 6. Terminological ambiguity, neighboring benchmarks, and interpretation

The label **LCB-Pro-Dafny** should be interpreted with care. In the AxDafny paper, it denotes the **250**-problem competition-style benchmark just described [2606.32007]. In "A benchmark for vericoding: formally verified program synthesis," however, the label refers to the Dafny component of a larger multilingual benchmark containing **3,029 Dafny specs** out of **12,504 formal specifications** overall, with **6,174** new unseen problems [2509.22908]. That broader benchmark reports a **model union** Dafny success rate of **82.2%**, while also noting a separate improvement on original DafnyBench verification from **68%** to **96%** for the model union over roughly a year [2509.22908]. The two uses are compatible in theme but not identical in benchmark definition, size, or task construction.

A second common confusion concerns **Multi-LCB**. Although it extends LiveCodeBench to multiple programming languages, its supported set is explicitly **C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin, and Scala**, and **Dafny does not appear anywhere in the benchmark language list, the tables, the runtime/compiler environment section, or the evaluation results** [2606.20517]. Multi-LCB is therefore not an implementation of LCB-Pro-Dafny, though its contamination-aware design and format-conversion methodology are relevant as a blueprint for possible future extensions.

The broader vericoding literature also sharpens how LCB-Pro-Dafny should be understood conceptually. The multilingual vericoding benchmark argues that formally verified synthesis is a more rigorous alternative to natural-language-only code generation because the goal is not merely plausible code but code whose correctness is formally established against a human-written specification [2509.22908]. In a separate experiment on Verina, that paper found that adding informal descriptions provided **no statistically significant performance improvement** and was slightly worse on average. This suggests that, for benchmarks such as LCB-Pro-Dafny, the decisive artifact is the formal specification rather than auxiliary natural-language description.

Taken together, these benchmark families indicate a shift in Dafny evaluation from **proof-hint reconstruction** toward **verified program synthesis**, and from verifier-only success toward a more differentiated view in which specification fidelity, anti-cheating validation, and executable behavior under resource limits are treated as distinct dimensions of performance.

Source: https://www.emergentmind.com/topics/livecodebench-pro-dafny-lcb-pro-dafny