AxDafny: Agentic Verified Code Generation in Dafny

Abstract: We study agentic code generation in Dafny, where a model must generate both executable code and the proof artifacts for verification. We present AxDafny, a verifier-guided repair framework that iteratively generates implementations, invariants, assertions, and termination arguments. We also introduce LiveCodeBench-Pro-Dafny (LCB-Pro-Dafny), a benchmark of 250 competition-style programming problems translated into Dafny with formal specifications and a verifier-based evaluation harness. On LCB-Pro-Dafny, AxDafny substantially improves verification success over baseline GPT-5.5 performance. On DafnyBench, AxDafny achieves 92.7\% verification success, outperforming the strongest previously reported proof-hint baseline by 6.5 percentage points. Lastly, we show that verification success and runtime test performance measure different aspects of generated code.

• The paper presents AxDafny, which jointly synthesizes Dafny code and its formal proofs using an iterative generate–check–repair loop.
• It introduces the LCB-Pro-Dafny benchmark, converting 250 competition-style problems into Dafny tasks with graded difficulties.
• Empirical results show substantial improvements over baselines in proof-hint and end-to-end synthesis, despite challenges with resource constraints.

AxDafny: Agentic Verified Code Generation in Dafny

Problem Formulation and Motivation

AxDafny addresses the challenge of agentic code generation in Dafny, focusing not only on synthesizing executable code, but also the nontrivial proof artifacts required for formal verification. Prior systems largely centered on proof annotation or hint regeneration for fixed implementations, often leveraging retrieval from curated libraries. In contrast, AxDafny targets joint synthesis of implementation and proof structure, requiring models to generate loop invariants, assertions, and termination arguments in addition to code. The paper introduces a new benchmark, LCB-Pro-Dafny, translating 250 competition-style programming problems into Dafny with formal specifications. By imposing a rigorous verification-oriented acceptance criterion, AxDafny studies the efficacy and limitations of LLM-driven verifier-guided repair loops in producing machine-verified, functionally correct software.

Agentic Architecture and Benchmarks

AxDafny implements a generate–check–repair loop, adapting the proposer–reviewer–memory paradigm from previous work in automated theorem proving. The proposer agent synthesizes full Dafny programs, while the reviewer applies multi-stage deterministic and LLM-based checks: preservation of specification strength, filtering of vacuous proof shortcuts such as assume or {:verify false}, and rejection of trivially true predicates via LLM review. Verifier feedback—including counterexamples and diagnostic traces—drives iterative repair. Memory is structured via self-managed reflection, relaying condensed lessons across attempts.

Evaluation is performed on two tasks:

• DafnyBench: Standard proof-hint synthesis, where code and specification are fixed and the agent supplies missing annotations.
• LCB-Pro-Dafny: End-to-end program synthesis with formal specifications, requiring generation of both implementation and proofs.

LCB-Pro-Dafny consists of 250 curated tasks stratified by difficulty (100 easy, 100 medium, 50 hard), enabling granular assessment of synthesis performance and benchmark coverage.

Empirical Results

DafnyBench: Proof-Hint Synthesis

AxDafny achieves verified pass rates of 92.7% (Gemini-3.1-Pro) and 88.9% (GPT-5.5 medium) on the established DafnyBench task, outperforming the strongest previously reported baseline, DafnyPro, by 6.5 percentage points. Notably, AxDafny surpasses retrieval-augmented systems without leveraging library hint augmentation. Gains are concentrated in the earliest iterations of the agentic loop. By iteration 5, primary feedback improvements saturate for all models.

Figure 1: Cumulative AxDafny pass rate on DafnyBench vs. iteration budget across model settings. The primary feedback gains are heavily concentrated within the first 5 iterations.

LCB-Pro-Dafny: End-to-End Program Synthesis

On LCB-Pro-Dafny, AxDafny verifies 75.0% of easy, 52.0% of medium, and 28.0% of hard instances—56.4% overall—representing a substantial improvement over direct GPT-5.5 pass@1 baseline (11.6% overall). Accuracy scales steadily with iterative budget, especially for harder instances, contrasting with the more concentrated improvement seen in proof-hint tasks. End-to-end synthesis derives more sustained benefits from multiple inference passes.

Figure 2: Cumulative AxDafny pass rate on LCB-Pro-Dafny across difficulty splits. Accuracy scales steadily with increased iteration budget across all levels. End-to-end synthesis benefits more from extended iterations than proof-hint tasks.

Model Ablation

Performance is model-sensitive: on the easy split of LCB-Pro-Dafny, GPT-5.5 medium verifies 86/100, Gemini-3.1-Pro 77/100, GPT-5.5 low 75/100, and Claude Opus 4.5 52/100, with verified pass rates reflecting downstream impact of base model reasoning capabilities and inference configuration.

Figure 3: Model ablation on the LCB-Pro-Dafny easy split. Bars report verified pass rate over 100 easy instances.

Executable Correctness and Resource-Limited Failures

Verified Dafny programs are compiled to Python and evaluated under LiveCodeBench-Pro’s original test harness. Among verified solutions, most failures are due to resource limits (TLE/MLE), not functional incorrectness. This divergence is attributable to Dafny’s focus on functional correctness, with specifications seldom constraining asymptotic complexity or memory usage. Additionally, Dafny’s compilation layer introduces potential runtime inefficiencies relative to native Python or C++.

Implications and Limitations

AxDafny demonstrates that verifier-guided repair substantially advances both proof annotation and program synthesis in Dafny. The strong numerical results for iterative agentic refinement—particularly in proof-hint synthesis—affirm the value of formal methods as a rich feedback source for LLM-assisted coding. However, the gap between formal verification and runtime test performance persists: many verified implementations fail under competitive programming constraints due to inefficiency or resource exhaustion. This highlights the need for future systems to incorporate complexity and resource reasoning within specification and synthesis.

Another structural limitation is the scarcity of high-quality, scalable Dafny training data compared to mainstream languages. GitHub statistics indicate orders-of-magnitude lower public code volume, constraining potential for language-specific LLM tuning and retrieval-based augmentation.

Prospects for Future Research

The introduction of LCB-Pro-Dafny offers a challenging and scalable benchmark for verified program synthesis, shifting evaluation targets towards joint implementation–proof generation. Future work may refine synthesis approaches by:

• Integrating explicit complexity/memory constraints in specifications.
• Enhancing agent memory via more sophisticated reflection and provenance.
• Extending verifier-guided synthesis to other formal languages (e.g., Lean, Isabelle/HOL).
• Exploring cross-language transfer and multimodal retrieval for improved annotation and implementation coverage.
• Advancing specification generation, validation, and diagnosis for robust formal verification in diverse settings.

Conclusion

AxDafny executes a rigorous study of agentic code generation in Dafny, using verifier-guided iterative repair to synthesize both implementation and formal proof. The system surpasses prior baselines in proof-hint synthesis and establishes substantial gains in end-to-end program synthesis, although strong limitations remain on competitive-resource benchmarks. The work motivates further investigation of scalable formal verification, resource-aware synthesis, and transfer learning for code generation in under-resourced languages, leveraging liquid agentic architectures and rich verifier feedback loops.