LCB-Pro-Dafny: Verified Synthesis Benchmark
- LCB-Pro-Dafny is a benchmark for end-to-end verified program synthesis in Dafny, featuring 250 competition-style tasks with natural language, method signatures, and formal specifications.
- It evaluates systems on synthesizing both executable code and accompanying proof artifacts that must verify against stringent Dafny contracts without relying on proof bypasses.
- The benchmark emphasizes verifier-guided synthesis over mere proof repair, with tasks categorized by difficulty (easy, medium, hard) to measure improvements in formal verification and functional correctness.
Searching arXiv for the named benchmark and closely related Dafny verification benchmarks to ground the article in current papers. LCB-Pro-Dafny is a benchmark for end-to-end verified program synthesis in Dafny built from 250 competition-style programming problems from LiveCodeBench-Pro. Each task presents the original natural-language problem statement, a Dafny method signature, and a formal specification, and the objective is not merely to produce runnable code but to generate both executable code and proof artifacts so that the solution verifies in Dafny against the formal contract. The benchmark was introduced together with AxDafny in “AxDafny: Agentic Verified Code Generation in Dafny” (Breen et al., 30 Jun 2026), where it is positioned as a testbed for verifier-guided program synthesis rather than ordinary runtime-only code generation.
1. Scope and benchmark identity
LCB-Pro-Dafny was introduced to evaluate verified synthesis in a setting that is broader than proof repair. The motivating distinction is central: existing Dafny benchmarks had largely emphasized proof-hint synthesis, whereas LCB-Pro-Dafny asks a model to synthesize the implementation and the proof structure directly from a problem statement. In that sense, it targets the full pipeline of turning an informal programming task into a machine-checked Dafny artifact (Breen et al., 30 Jun 2026).
The benchmark inherits its instances from LiveCodeBench-Pro and retains the original difficulty split:
| Split | Tasks |
|---|---|
| Easy | 100 |
| Medium | 100 |
| Hard | 50 |
This design matters because the benchmark is intended to be larger, more realistic, and more challenging than earlier Dafny synthesis datasets. The paper contrasts it with DafnyBench, which focuses on restoring missing invariants or assertions in already-verified code, and with earlier synthesis benchmarks such as Dafny-synthesis with 178 tasks from MBPP and HumanEval-Dafny with 129 tasks, which are described as limited by scale, simplicity, contamination concerns, and the lack of runtime constraints (Breen et al., 30 Jun 2026). A plausible implication is that LCB-Pro-Dafny is meant to move Dafny evaluation closer to competitive-programming-style synthesis while preserving formal verification as the primary criterion.
2. Task construction and specification curation
The benchmark construction was treated as a curation process rather than a fully automatic translation step. According to the paper, the initial Dafny specifications were produced with model assistance, then manually reviewed. During that review, the authors removed or revised tasks with inconsistent, underspecified, or vacuous specifications, and ensured that the remaining specifications captured the intended functional behavior of the original problems (Breen et al., 30 Jun 2026).
The final release contains 250 selected instances, and all released specifications parse and type-resolve under Dafny 4.11.0. The appendix reports that under dafny resolve --allow-warnings, all 250 specifications parse and type-resolve; no missing ensures clauses or trivially true postconditions were found by static checks; and under Dafny’s stricter default warning policy, 49 specifications emitted quantifier-trigger warnings but otherwise had no errors. An additional review layer used LLM-as-judge analysis: 14/250 specifications (5.6%) were flagged for semantic mismatches requiring fixes, and all flagged cases were revised before inclusion (Breen et al., 30 Jun 2026).
The paper also describes explicit vacuity checks. No placeholder implementation verified, and no automatically generated constant-or-empty-output mutation verified in the 234 cases where the mutation harness applied. A representative example involved a set of chosen objects represented as a sequence without forbidding duplicates; the fix was to require the sequence to be distinct. These checks are significant because they address a standard concern in verification benchmarks: a formally checkable specification can still be too weak to encode the intended problem semantics. Here, the reported validation process was designed to reduce that risk without claiming fully automatic soundness of the benchmark curation.
3. Evaluation protocol and acceptance semantics
LCB-Pro-Dafny uses a verifier-based harness rather than runtime tests alone. For program synthesis on this benchmark, a solution is accepted only when (1) the generated implementation verifies in Dafny, (2) the provided specification, definitions, and functions are not modified, and (3) the solution does not rely on vacuous proof shortcuts, including assume or {:extern}. The benchmark also rejects proof-bypass constructs such as {:axiom} and {:verify false} (Breen et al., 30 Jun 2026).
This acceptance protocol is conceptually important. In standard coding benchmarks, success is defined operationally by passing test suites. In LCB-Pro-Dafny, success is defined semantically by proving the generated Dafny program against an explicit formal contract. The verifier therefore acts as the primary arbiter, and additional checks enforce that verification is not achieved by specification mangling or proof bypass. This makes the benchmark evaluator closer to a proof checker than to a standard execution harness.
The benchmark nevertheless includes a secondary executable-correctness evaluation. Verified Dafny solutions are compiled to Python and run against the original LiveCodeBench-Pro test suite under the original time limits. This second measure is separate from verification acceptance and is included precisely to compare formal verification success with ordinary executable performance (Breen et al., 30 Jun 2026). The distinction is not cosmetic: the paper explicitly argues that the two metrics capture different properties.
4. Measured difficulty and reported results
On LCB-Pro-Dafny, AxDafny verifies 56.4% overall of the 250 tasks, while direct GPT-5.5 pass@1 achieves 11.6% overall. Broken down by difficulty, AxDafny reaches 75.0% on easy (75/100), 52.0% on medium (52/100), and 28.0% on hard (14/50). GPT-5.5 pass@1 reaches 13.0% on easy, 14.0% on medium, and 4.0% on hard (Breen et al., 30 Jun 2026).
| System | Verification results |
|---|---|
| AxDafny | Easy 75.0%, Medium 52.0%, Hard 28.0%, Overall 56.4% |
| GPT-5.5 pass@1 | Easy 13.0%, Medium 14.0%, Hard 4.0%, Overall 11.6% |
The paper interprets these results as evidence that verifier-guided iterative repair substantially improves verified Dafny synthesis relative to one-shot direct generation. At the same time, the hard split remains explicitly “far from saturated,” and the reported 28.0% hard-split result is presented as evidence that LCB-Pro-Dafny is a challenging benchmark for end-to-end verified program synthesis (Breen et al., 30 Jun 2026). This suggests that verifier feedback materially improves synthesis, but does not remove the underlying difficulty of generating implementations, invariants, assertions, and termination arguments for competition-style tasks.
The same paper reports that AxDafny also achieves 92.7% verification success on DafnyBench, outperforming the strongest previously reported proof-hint baseline by 6.5 percentage points (Breen et al., 30 Jun 2026). The contrast is instructive: strong results on proof-repair-oriented benchmarks do not imply comparable performance on LCB-Pro-Dafny, whose task definition is broader.
5. Verification success versus executable success
A recurring theme in the benchmark design is that verification acceptance is not equivalent to benchmark success under traditional competitive-programming metrics. The verifier checks functional correctness against explicit specifications, but the paper notes that those specifications generally do not constrain asymptotic complexity or memory usage. As a result, a verified solution can still perform poorly under the original LiveCodeBench-Pro runtime harness (Breen et al., 30 Jun 2026).
The secondary executable results make this point concrete. On the easy split, AxDafny verifies 75/100 tasks; among those verified outputs, 32 pass the original executable tests, 39 fail by time limit exceeded (TLE), and 4 fail by memory limit exceeded (MLE). On the medium split, AxDafny verifies 52/100 tasks; among those, 6 pass, 44 fail by TLE, and 2 fail by MLE (Breen et al., 30 Jun 2026). The paper emphasizes that the failures are overwhelmingly resource-limit failures rather than wrong-answer failures.
This pattern is partly attributed to the difference between the formal contracts and the runtime environment. The specifications establish functional behavior but usually do not encode efficiency requirements, and Dafny-to-Python compilation can introduce additional overhead because runtime implementations for constructs such as sets and maps may be slower than native Python or C++. The authors report a lightweight Python compatibility layer as a partial mitigation, but they treat the distinction between proof obligations and resource constraints as fundamental rather than incidental (Breen et al., 30 Jun 2026).
One common misconception is therefore that “verified” means “competitive-programming ready.” LCB-Pro-Dafny is specifically designed to show that these notions diverge. Verification certifies conformance to the formal specification; it does not, by itself, certify suitability under the original benchmark’s time and memory limits.
6. Relation to the broader Dafny benchmark landscape
LCB-Pro-Dafny sits within a rapidly developing ecosystem of Dafny benchmarks, but its target differs from adjacent efforts. In “Dafny as Verification-Aware Intermediate Language for Code Generation,” Dafny is used as a verification-aware intermediate representation between a natural-language request and target-language code, with an iterative verifier-in-the-loop loop and final compilation to Python (Li et al., 10 Jan 2025). That work shares the idea that verification can act as a hidden correctness scaffold, but LCB-Pro-Dafny turns the emphasis from pipeline architecture to benchmark design: it evaluates whether a system can synthesize verified Dafny solutions for competition-style tasks rather than merely demonstrate a verification-mediated code-generation prototype.
In “Inferring multiple helper Dafny assertions with LLMs,” DAISY addresses missing helper assertions in failing Dafny programs and evaluates repair on curated variants of DafnyBench (Silva et al., 31 Oct 2025). That line of work is closely related but narrower. It assumes an existing program and proof context, whereas LCB-Pro-Dafny asks for synthesis of the implementation and proof structure from the problem statement itself. The distinction clarifies why the LCB-Pro-Dafny paper explicitly positions the benchmark against proof-hint synthesis.
“Local Success Does Not Compose: Benchmarking LLMs for Compositional Formal Verification” extends the difficulty frontier in another direction by introducing DafnyCOMP, a benchmark of 300 automatically synthesized multi-function programs where the task is to regenerate interprocedural contracts such as requires, ensures, reads, modifies, and decreases (Xu et al., 27 Sep 2025). That benchmark is diagnostic for compositional reasoning across function boundaries. Relative to LCB-Pro-Dafny, this suggests a broader research trajectory: isolated verified synthesis, proof repair, and compositional contract generation are distinct but connected evaluation regimes, and success in one does not necessarily transfer to the others.
Within that landscape, LCB-Pro-Dafny occupies a specific niche. It is neither a pure proof-repair corpus nor a purely compositional contract-reconstruction benchmark. Instead, it is a formally specified, verifier-centered benchmark for program synthesis, intended to push models beyond “passes tests” toward “provably correct under a specification,” while still quantifying how often such verified code also survives the original LiveCodeBench-Pro runtime harness (Breen et al., 30 Jun 2026).