miniF2F-Dafny Benchmark

• The paper presents a translation of the miniF2F benchmark into Dafny, enabling SMT-driven auto-active verification for olympiad-style math problems.
• It details a systematic conversion process and a structured pipeline that separates low-level SMT automation from high-level LLM-guided proof hints.
• Empirical results show nearly 56% success with hybrid verification, highlighting the effectiveness of combining strategic LLM guidance with automated SMT checks.

Searching arXiv for the specified paper and the original miniF2F benchmark to ground the article in published sources. miniF2F-Dafny is a translation of the mathematical reasoning benchmark miniF2F to the automated theorem prover Dafny, presented as “the first translation of the mathematical reasoning benchmark miniF2F to an automated theorem prover: Dafny” (Baksys et al., 11 Dec 2025). The benchmark repurposes 488 “olympiad-style” problems—244 test and 244 validation—originally formalized in interactive theorem provers (ITPs) such as Lean, Isabelle, HOL Light, and Metamath, into an auto-active verification setting in which the heavy lifting is delegated to Dafny and its SMT backend rather than to explicit proof scripts (Baksys et al., 11 Dec 2025). Its central research question is whether olympiad-style mathematical statements can be discharged either directly by SMT-driven automation or with limited high-level guidance from LLMs, rather than by generating fully elaborated proof terms.

1. Position within formal reasoning

Automated formal reasoning has “traditionally divided into two camps”: interactive theorem provers, “where users write explicit proof scripts,” and auto-active verifiers such as Dafny, “where the heavy lifting is done by SMT backends and users supply only guiding annotations” (Baksys et al., 11 Dec 2025). miniF2F had already “become a de facto standard for measuring progress in AI + formal mathematics within ITPs,” but its earlier incarnations required “fully fleshed-out proof terms and extensive user (or AI) intervention” (Baksys et al., 11 Dec 2025).

miniF2F-Dafny redefines that benchmarking setting. Rather than evaluating end-to-end proof-script synthesis, it asks two questions: “How many olympiad-style facts can Z3-driven automation discharge out of the box?” and, when that fails, “can off-the-shelf LLMs provide just enough high-level guidance—lemmas, proof strategies, intermediate assertions—so that Dafny’s SMT solver closes the remaining proof obligations?” (Baksys et al., 11 Dec 2025). This establishes a distinct methodological paradigm: from whole-proof generation to what the paper characterizes as “proof hints plus SMT” (Baksys et al., 11 Dec 2025).

This suggests that miniF2F-Dafny is not merely a format conversion of miniF2F, but a benchmark for a different proof-production regime. In that regime, strategic decomposition and local proof search are separated: the model proposes structure, while the verifier discharges low-level entailment.

2. Corpus and translation pipeline

The source benchmark “comprises 488 ‘olympiad-style’ problems (244 test, 244 validation) drawn from AMC/AIME/IMO contests and undergraduate textbooks” (Baksys et al., 11 Dec 2025). In miniF2F-Dafny, “each miniF2F problem originally formalized in Lean, Isabelle, HOL Light or Metamath is mechanically converted into a Dafny lemma” of the form:

include "../definitions.dfy" include "../library.dfy" lemma problem_name(params…) requires … // preconditions ensures … // postconditions { }

Two design decisions structure this translation (Baksys et al., 11 Dec 2025). First, the system axiomatizes “core mathematics in two files—definitions.dfy (integers, rationals, reals, complex) and library.dfy (108 standard lemmas in algebra, number theory, analysis, etc.).” Each function is “declared with {:axiom} and a contract of ensures clauses” (Baksys et al., 11 Dec 2025). The paper gives the following representative encoding of a sum-over-set operator:

function {:axiom} sum<T>(s: set<T>, f: T->real): real ensures s == {} ==> sum(s,f) == 0.0 ensures forall x | x in s :: sum(s,f) == f(x) + sum(s - {x},f)

Second, the translation is designed to “preserve semantics”: “preconditions (requires) and postconditions (ensures) are extracted directly from the ITP source via syntax-guided translation, then type-checked in Dafny” (Baksys et al., 11 Dec 2025). Each translated lemma begins with an empty proof body, written {}. The explicit purpose is to quantify “exactly how much can be solved by SMT alone, and where LLM guidance is truly needed” (Baksys et al., 11 Dec 2025).

The paper illustrates the translation with a basic arithmetic identity:

lemma plus_zero(x: nat) ensures x + 0 == x { }

The example is simple, but its significance is methodological: the same representation is applied to olympiad-style benchmark items, thereby enabling direct measurement of the contribution of automation before any LLM intervention.

3. Auto-active verification in Dafny

Dafny is described as combining “a rich specification language (pre/postconditions, ghost code, calc blocks) with the Z3 SMT solver” (Baksys et al., 11 Dec 2025). In this setting, an “empty proof” means that a lemma body is left as {}. When “the postconditions follow from the preconditions plus the imported library axioms,” Dafny marks the lemma verified without further proof steps (Baksys et al., 11 Dec 2025).

On miniF2F-Dafny, this empty-proof baseline verifies:

Split	Verified with empty bodies	Rate
Test set	99/244	40.6%
Validation set	109/244	44.7%

These numbers are reported directly for the translated benchmark (Baksys et al., 11 Dec 2025). The paper interprets them as showing that “nearly half of olympiad-level claims are within reach of off-the-shelf SMT automation, without any user- or model-supplied proof steps” (Baksys et al., 11 Dec 2025).

A common misconception would be to equate these results with fully autonomous theorem discovery. The benchmark instead measures verification of already formalized statements under a fixed axiomatized environment and imported lemma library. The automation is strong, but it operates within the problem specification and background theory established by definitions.dfy and library.dfy. This suggests that the reported success rates reflect both the inherent reach of SMT-backed auto-active verification and the expressivity of the chosen axiomatization.

4. LLM-guided proof hints

When empty proofs fail, miniF2F-Dafny evaluates “12 off-the-shelf LLMs, accessed via AWS Bedrock, to fill in proof-hint bodies” (Baksys et al., 11 Dec 2025). Each model is prompted with “the contents of definitions.dfy and library.dfy,” “the incomplete lemma stub,” “a system instruction forbidding assume or new {:axiom} claims,” and “encouragement to use Dafny idioms (assert, calc, helper lemmas)” (Baksys et al., 11 Dec 2025).

The evaluation uses “an iterative error-correction strategy: up to $N = 4$ generation attempts per problem, each followed by up to $E = 3$ rounds of error feedback” (Baksys et al., 11 Dec 2025). With $S$ the number of problems solved in the top $k$ attempts, the paper defines $\mathrm{pass@k} = S / \mathrm{total}$ (Baksys et al., 11 Dec 2025). On the 244-problem test split, the reported pass rates are:

Model	# problems	pass@1 (%)	pass@2 (%)	pass@4 (%)
Claude Sonnet 4.5	244	52.5	53.7	55.7
Claude Sonnet 4.0	243	49.7	50.9	53.0
Qwen 3 235B MoE	164	47.0	53.4	54.3
Dafny Verifier (no LLM)	244	40.6	40.6	40.6

The paper states that “the best model, Claude Sonnet 4.5, attains pass@4 = 55.7%, a 15.1-point absolute (≈37% relative) lift over the empty-proof baseline” (Baksys et al., 11 Dec 2025). It further attributes the gain to “multiple attempts and error-correction,” which allow the model to respond to SMT diagnostics by “reordering assertions, refining calc steps, or introducing missing helper lemmas” (Baksys et al., 11 Dec 2025).

The intended role of the model is therefore sharply delimited. Rather than replacing the verifier, it supplies “strategic proofs” in the form of proof hints. A plausible implication is that the benchmark probes a hybrid competence: syntactic fluency in Dafny, heuristic selection of intermediate claims, and responsiveness to verifier feedback.

5. System architecture and proof workflow

The paper presents miniF2F-Dafny as an orchestrated pipeline with four stages (Baksys et al., 11 Dec 2025). First, “Translation” produces “a directory of 488 Dafny stubs + shared library files.” Second, the “Empty-proof stage” invokes dafny “on each stub (30 s timeout, up to 5 retries)” and records passes and failures. Third, “LLM hint generation (for fails)” proceeds by sending the initial prompt to the model, verifying the returned body with Dafny, and, if verification fails, feeding the resulting SMT errors back into an error-correction prompt. Fourth, “Validation” rejects any solution that “weakens requires or ensures, or uses banned constructs” (Baksys et al., 11 Dec 2025).

The paper summarizes the procedure with the following pseudocode:

for lemma in miniF2F-Dafny: if verify(lemma) succeed: record baselinesuccess else: for attempt in 1..N: hint_body = LLM.generate(prompt(lemma, history)) lemma_filled = lemma.withBody(hint_body) if verify(lemma_filled) succeed: record llmsuccess break else: history.append(smtErrors)

This architecture “cleanly decouples” two components: “the SMT-centric proof engine—responsible for low-level entailment” and “the LLM proof ‘hinter,’ which focuses on high-level reasoning patterns: choosing the right lemma, structuring calc proofs, or applying classical inequalities” (Baksys et al., 11 Dec 2025). The paper’s recurring phrase, “division of labor,” captures this decomposition.

That division is also evaluative rather than merely implementational. Because every problem first passes through the empty-proof stage, the benchmark distinguishes what can already be discharged by the background theory and solver from what additionally requires model-generated annotations.

6. Limitations, trade-offs, and prospective directions

The paper identifies three “key insights and trade-offs” (Baksys et al., 11 Dec 2025). First, “auto-active verification handles a surprisingly high fraction (≈40%) of olympiad problems out-of-the-box.” Second, “off-the-shelf LLMs need only supply ‘strategic proofs’ rather than line-by-line scripts, boosting pass rates to ≈56%.” Third, “the iterative feedback loop—feeding SMT errors back to the model—yields substantial gains from a single model” (Baksys et al., 11 Dec 2025).

It also lists three limitations. “Verification brittleness” means that “small changes in assert order or missing triggers on quantifiers can derail Z3” (Baksys et al., 11 Dec 2025). In addition, “LLMs are not fully fluent in Dafny syntax and often conflate ghost code or calc constructs with Lean idioms” (Baksys et al., 11 Dec 2025). Finally, “the minimal library (550 lines vs. Mathlib’s 2 M) means many proofs require new theory development, which LLMs sometimes fail to propose” (Baksys et al., 11 Dec 2025).

These caveats delimit the significance of the reported pass rates. The benchmark does not claim that olympiad mathematics is broadly solved in Dafny; rather, it demonstrates a measurable interaction between a compact axiomatized library, SMT automation, and LLM-supplied proof scaffolding. The comparison “550 lines vs. Mathlib’s 2 M” is especially important context, because it indicates that the benchmark operates under a substantially leaner background theory than large ITP ecosystems (Baksys et al., 11 Dec 2025).

The future directions proposed in the paper follow directly from these bottlenecks (Baksys et al., 11 Dec 2025). They include “Dafny-specific fine-tuning,” namely adapting LLMs “on large corpora of Dafny-verified methods to internalize idioms like calc or triggers”; “agentic orchestration,” with a “multi-component proof agent that interleaves lemma synthesis, counterexample-guided refinement, and library expansion”; and “learned lemma libraries,” where the system can “extract, generalize, and cache successful proof patterns as new lemmas” (Baksys et al., 11 Dec 2025). This suggests that miniF2F-Dafny may serve both as a benchmark and as a substrate for research on auto-active proof agents.

7. Significance for benchmarking AI-assisted mathematics

miniF2F-Dafny is defined by a benchmarking shift rather than only by a new corpus format. Previous miniF2F incarnations existed only in “interactive theorem provers (Lean, Isabelle, HOL Light, Metamath),” whereas this benchmark relocates the same class of problems into an “auto-active setting” (Baksys et al., 11 Dec 2025). Its reported results support the paper’s claim of “an effective division of labor: LLMs provide high-level guidance while automation handles low-level details” (Baksys et al., 11 Dec 2025).

The benchmark is therefore significant for at least three reasons grounded in the paper’s presentation. First, it provides a direct empirical estimate of what SMT-backed automation can already prove with no manual proof steps on olympiad-style problems. Second, it operationalizes a constrained role for LLMs: not theorem proving in isolation, but verifier-conditioned generation of annotations, helper lemmas, assertions, and calc structure. Third, it exposes a set of failure modes specific to this hybrid regime, including trigger sensitivity, annotation ordering, and syntactic mismatch between proof-assistant idioms and Dafny idioms (Baksys et al., 11 Dec 2025).

The benchmark is available on GitHub at http://github.com/dafny-lang/miniF2F (Baksys et al., 11 Dec 2025). In the paper’s own framing, miniF2F-Dafny “opens a new frontier”: it demonstrates that “an auto-active verifier, combined with modest LLM guidance, can tackle a broad class of olympiad-style mathematics with far less manual effort than traditional ITP frameworks” (Baksys et al., 11 Dec 2025). A cautious reading is that the benchmark does not eliminate the need for formal libraries, solver engineering, or proof design; instead, it recasts their interaction in a way that is measurable, modular, and amenable to iterative LLM-verifier loops.