MiniF2F-ALF: Robustness Perturbation Benchmark
- The paper introduces MiniF2F-ALF, a contamination-sensitive perturbation benchmark for Lean that generates two divergent mutations per original theorem.
- It employs an ALF mutation pipeline spanning five categories to create and select semantically divergent yet formally valid Lean statements.
- Empirical results show all evaluated provers lose accuracy under perturbations, highlighting the challenge of sustaining robustness against statement-level shifts.
Searching arXiv for the benchmark and closely related miniF2F work to ground the article in the relevant papers. {"query":"MiniF2F-ALF Pythagoras-Prover miniF2F v2 formal proving Lean", "max_results": 10} Searching for the Pythagoras-Prover paper and the miniF2F-v2 benchmark paper. arxiv_search: query: "Pythagoras-Prover Advancing Efficient Formal Proving via Augmented Lean Formalisation" max_results: 5 MiniF2F-ALF is a contamination-sensitive perturbation benchmark for Lean theorem proving obtained by mutating the 244 statements in MiniF2F-Test with Augmented Lean Formalisation (ALF) and retaining two divergent formal variants per source theorem, yielding 488 evaluation statements. It was introduced together with the Pythagoras-Prover family to measure robustness under statement-level distribution shift, with the central hypothesis that a prover with transferable proof competence should remain more stable under controlled formal perturbations than a prover that depends on memorized benchmark forms or benchmark-specific surface templates (Leang et al., 10 Jun 2026). Within the broader miniF2F lineage, it differs from correction-oriented revisions such as miniF2F-v2, which target discrepancies between informal and formal statements, because MiniF2F-ALF is not a correction release but a structured perturbation benchmark built from the test split itself (Ospanov et al., 5 Nov 2025).
1. Position within the miniF2F ecosystem
miniF2F is a benchmark of Olympiad-style mathematics problems used for autoformalization and theorem proving in Lean, and later work reframed it as an end-to-end “AI math Olympiad” pipeline in which a system reads an informal statement, translates it into Lean, proves it, and receives credit only if the resulting formal theorem and proof correspond to the original problem (Ospanov et al., 5 Nov 2025). In that setting, benchmark quality and theorem-statement fidelity become part of the evaluation problem itself.
MiniF2F-ALF occupies a different niche. It preserves the original MiniF2F-Test problem family but changes the statement form. The paper characterizes it as a contamination-sensitive perturbation benchmark rather than a new official split or a repair of defective items. It is intentionally distribution-shifted while remaining close to the original, and it is designed to expose whether strong MiniF2F-Test performance transfers to nearby formal variants or depends on narrow familiarity with specific theorem phrasings (Leang et al., 10 Jun 2026).
This distinction matters because the miniF2F literature identifies two separate evaluation pathologies. One is statement misalignment between informal and formal versions, emphasized by miniF2F-v2. The other is saturation and contamination sensitivity on MiniF2F-Test, emphasized by MiniF2F-ALF. The former concerns whether the benchmark theorem is faithful; the latter concerns whether a model remains reliable when a faithful theorem is reformulated.
2. ALF as a mutation operator and benchmark construction method
ALF denotes Augmented Lean Formalisation. In the broader Pythagoras-Prover framework, ALF is used to expand scarce verified corpora into variants of formal statements through a three-stage pipeline: statement mutation, proof self-distillation, and statement-alignment filtering (Leang et al., 10 Jun 2026).
The mutation stage defines five categories exactly: Simplification, Generalisation, Lemma proposal, Proof-step decomposition, and Reformulation. In the general ALF pipeline, a dedicated mutation model, Qwen3.6-27B, emits one variant in each category for a seed theorem statement. The self-distillation stage then uses the post-RL Pythagoras-Prover to generate a single proof candidate for each mutated statement, with per mutation. Filtering is performed through a statement-alignment check: a pair is kept only if the proof references the intended target statement. The paper explicitly states that this is not a correctness check and does not invoke Lean on every training instance.
MiniF2F-ALF uses the same mutation idea in benchmark form, but with stricter retention rules. For each original MiniF2F-Test theorem, the benchmark construction generates five candidate mutations with Codex (GPT-5.5), adds additional numerical and variable perturbation inspired by GSM-Symbolic, ranks the candidates by semantic divergence using cosine distance in the embedding space of Alibaba-NLP/gte-Qwen2-7B-instruct, and keeps the two most divergent ones. The resulting statements are intended to preserve formal well-formedness and near-semantic relation to the source while changing the surface form enough to stress the prover (Leang et al., 10 Jun 2026).
A recurrent design principle is that near-duplicate perturbations are not very informative. The benchmark therefore selects for divergence rather than merely for formal validity. This suggests that MiniF2F-ALF is structured to emphasize robustness to nontrivial formal paraphrase rather than robustness to cosmetic renaming alone.
3. Composition, verification, and scoring protocol
The benchmark contains 488 problems: the 244 original MiniF2F-Test statements together with two retained mutations per original problem. The paper distinguishes this from the ALF training corpus. In the training setting, exhaustive Lean verification is omitted for scale; in the benchmark setting, every retained mutation is checked to be a well-formed Lean theorem, passed through the Lean compiler, and verified with both a judge model and Lean itself (Leang et al., 10 Jun 2026).
Evaluation uses Lean 4.9.0-rc1 with a maximum generation length of 30,000 tokens. A generated proof is counted as correct only if two conditions hold. First, it compiles in Lean with no errors and contains no sorry, admit, or unresolved goals. Second, the target formal statement must appear verbatim in the generated output. The second condition is specifically intended to prevent a prover from solving a different, weaker, or silently rewritten theorem.
The reported metric is pass@, defined as the fraction of problems for which at least one of the sampled attempts succeeds:
This scoring rule makes MiniF2F-ALF a contamination-sensitive evaluation not by introducing a new scalar robustness metric, but by combining statement perturbation with the standard Lean correctness criterion and a verbatim-target constraint (Leang et al., 10 Jun 2026).
4. Empirical behavior under perturbation
The central empirical result is that every evaluated model loses accuracy on MiniF2F-ALF relative to MiniF2F-Test. The paper treats this as evidence that the perturbations induce a meaningful distribution shift rather than a trivial reformatting of the original benchmark (Leang et al., 10 Jun 2026).
The headline values emphasized in the paper are summarized below.
| Model | MiniF2F-Test | MiniF2F-ALF |
|---|---|---|
| Pythagoras-Prover-32B | 89.8 | 85.0 |
| Pythagoras-Prover-4B | 86.1 | 83.2 |
| Goedel-Prover-V2-32B | 88.1 | 83.6 |
Among these systems, Pythagoras-Prover-32B achieves the highest absolute pass rate on MiniF2F-ALF at 85.0, while Pythagoras-Prover-4B reaches 83.2 and is described as almost on par with Goedel-Prover-V2-32B at 83.6. The paper also states explicit degradation figures for two models: Pythagoras-Prover-4B degrades by 2.9 points, whereas Goedel-Prover-V2-32B degrades by 4.5 points (Leang et al., 10 Jun 2026).
The analysis does not provide a per-mutation-type accuracy table, but it does isolate a structural effect of the perturbation benchmark. On the original split, failures are concentrated in AMC and IMO. On MiniF2F-ALF, failures diversify and include more MathD, Algebra, Number Theory, Induction, and AIME items. This shift implies that MiniF2F-ALF surfaces brittleness on statements that are not necessarily intrinsically the hardest in the original benchmark, but become difficult once theorem form is altered (Leang et al., 10 Jun 2026).
5. Diagnostic role in training and generalization studies
MiniF2F-ALF serves not only as an evaluation benchmark but also as a train-test transfer probe for ALF-based data augmentation. The same ALF operator used to create mutated evaluation statements is used in Pythagoras-Prover training to expand formal corpora through self-distilled statement variants. The benchmark therefore tests whether exposure to structured formal mutations during training improves robustness to analogous mutations at evaluation time (Leang et al., 10 Jun 2026).
In that setting, the paper interprets the smaller degradation of ALF-trained Pythagoras models as evidence of improved robustness to theorem-statement variation. The claim is not that ALF eliminates the perturbation gap; all models still lose accuracy. Rather, the benchmark differentiates systems by how sharply they degrade when theorem form changes while theorem-like formal character is preserved.
This role becomes more significant because MiniF2F-Test is described as increasingly saturated. On a nearly saturated benchmark, raw pass rates can compress the distinction between genuine generalization and benchmark-specific familiarity. MiniF2F-ALF restores discrimination by introducing controlled shift without leaving the Lean theorem-proving domain. A plausible implication is that it functions as a robustness-oriented complement to ordinary MiniF2F-Test evaluation, especially for high-performing models whose baseline pass rates are already close.
6. Relation to benchmark validity debates and interpretive cautions
The miniF2F-v2 study argues that the original miniF2F benchmark contains substantial mismatches between informal and formal statements, with more than half of the problems exhibiting some discrepancy, and reports that such mismatches can sharply distort end-to-end pipeline accuracy (Ospanov et al., 5 Nov 2025). That critique targets semantic fidelity between contest problems and formal benchmarks.
MiniF2F-ALF addresses a different issue. It assumes retained evaluation items are valid Lean theorems and asks whether provers remain effective when valid theorem statements are perturbed. It therefore should not be read as a substitute for benchmark repair. Nor should it be read as merely a harder version of MiniF2F in the usual sense. The paper explicitly states that its purpose is not primarily to make problems harder in absolute terms, but to make them different in form while preserving formal character (Leang et al., 10 Jun 2026).
A second interpretive caution concerns verification. In the ALF training pipeline, mutated statement-proof pairs are filtered by statement alignment rather than exhaustive formal verification. In the benchmark, by contrast, the final retained statements are checked for well-formedness and passed through Lean. The distinction is technically important: MiniF2F-ALF as an evaluation set is compiler-validated, whereas ALF as a corpus-construction method is intentionally cheaper and does not certify every generated training example.
Taken together, these features make MiniF2F-ALF a benchmark for robustness to formal restatement, contamination sensitivity, and statement-level transfer. It complements, rather than replaces, the concerns raised by miniF2F-v2 about informal-formal alignment. In the current miniF2F ecosystem, the two lines of work target different failure modes: one addresses whether the theorem being scored is faithful to the original mathematics, and the other addresses whether a prover’s success survives controlled perturbation of the theorem statement.