ConsistencyCheck: Semantic Fidelity Benchmark

• ConsistencyCheck is a benchmark that evaluates whether formal Lean statements semantically match their natural-language math problems, focusing on semantic consistency beyond mere syntactic validity.
• It is built from 859 expert-annotated items using rigorous protocols and binary labels to detect subtle semantic errors such as quantifier scope and boundary conditions.
• Empirical evaluations show models like Gemini-2.5-Pro achieving around 85.8% accuracy, highlighting the persistent challenges of maintaining semantic fidelity in autoformalization.

ConsistencyCheck is a benchmark introduced to evaluate whether LLMs can reliably judge semantic correctness in autoformalization, the task of determining whether a formal Lean statement preserves the meaning of an original natural-language mathematics problem. It was introduced alongside ReForm and Prospective Bounded Sequence Optimization (PBSO) because evaluation in autoformalization had become overly dependent on LLM judges without sufficiently testing whether those judges are themselves trustworthy (Chen et al., 28 Oct 2025). In this setting, the central issue is not proof validity or syntactic well-formedness alone, but semantic consistency: whether a compilable Lean statement truly means the same thing as the source problem. ConsistencyCheck therefore serves both as a benchmark for semantic judges and as empirical evidence that even expert-written formalizations are far from trivial (Chen et al., 28 Oct 2025).

1. Definition and evaluative role

ConsistencyCheck is the paper’s dedicated benchmark for evaluating whether LLMs can reliably judge semantic correctness in autoformalization (Chen et al., 28 Oct 2025). The benchmark is motivated by the observation that the paper’s main metric is semantic consistency, and that reported gains would be difficult to interpret if the judge itself were noisy.

The benchmark adopts a binary notion of correctness. Given a natural-language problem $Q$ and a formal Lean statement $\text{Ans}$, the statement is correct only if it is both compilable in Lean and semantically faithful to $Q$. This is the same semantic notion used in the paper’s main evaluation. The task is therefore not proof correctness, but whether the formal statement means the same thing as the problem (Chen et al., 28 Oct 2025).

In the paper’s training setup, this notion is encoded by the reward

$r_{\text{task}(Q,\text{Ans}) = \begin{cases} 1 & \text{if } \text{PassesLean}(\text{Ans}) \land \text{IsConsistent}(Q,\text{Ans}) \ 0 & \text{otherwise} \end{cases}$

where $\text{IsConsistent}$ is a binary semantic judge. ConsistencyCheck is the benchmark used to validate that this kind of judge is sufficiently reliable (Chen et al., 28 Oct 2025).

A central implication is that ConsistencyCheck functions as both a measurement instrument and an argument about evaluation methodology. The benchmark is used to test whether semantic judgment is itself reliable enough to support claims about progress in autoformalization, rather than merely assuming that a judge’s outputs are accurate.

2. Construction and annotation protocol

ConsistencyCheck is constructed from 859 expert-annotated items drawn from miniF2F and ProofNet (Chen et al., 28 Oct 2025). The annotation process is intentionally rigorous. Expert annotators with deep proficiency in mathematics and Lean4 first inspect anonymized formal statements paired with the original problem text. Two experts independently label each item as either “Correct” or “Incorrect” depending on whether the formalization faithfully captures the mathematical intent. When they disagree, a third senior expert adjudicates. If an item is marked incorrect, annotators also provide written justification for the semantic mismatch (Chen et al., 28 Oct 2025).

The benchmark is explicitly not a syntactic check. Its focus is on subtle semantic fidelity issues, including quantifier scope, hidden constraints, wrong constants, boundary-condition drift, or changing the meaning of the original problem (Chen et al., 28 Oct 2025). The included examples illustrate such pitfalls, including replacing $\sqrt{11}$ with $11$, changing “$\le 80$” to “$<80$,” or adding an explicit answer where the original problem did not (Chen et al., 28 Oct 2025).

This annotation design makes the benchmark stricter than compiler-based validation. A Lean statement may compile and still be judged incorrect if it introduces semantic drift relative to the source problem. This suggests that semantic auditing must be treated as a distinct layer of evaluation rather than as a by-product of syntactic verification.

3. Semantic scope and error typology

The semantic object measured by ConsistencyCheck is narrow in one sense and demanding in another. It is narrow because it asks a binary question: does the formal Lean statement preserve the meaning of the natural-language problem? It is demanding because the errors of interest are often small local edits with global semantic effect (Chen et al., 28 Oct 2025).

The benchmark emphasizes exactly the kinds of errors that can evade compiler-based validation. These include hidden semantic dimensions such as quantifier scope, implicit constraints, boundary conditions, and whether the statement introduces or omits conclusions not present in the source problem (Chen et al., 28 Oct 2025). The paper’s examples make clear that semantically incorrect formalizations may still be syntactically acceptable and machine-checkable.

This focus places ConsistencyCheck in a broader family of research that treats “consistency checking” as a task distinct from surface validity. In factual consistency evaluation, WeCheck similarly treats the problem as deciding whether a hypothesis is factually consistent with its premise rather than whether it is merely fluent (Wu et al., 2022). In LLM development, SimCT defines consistency as whether models at different stages remain behaviorally equivalent in their decoding distribution, rather than whether their outputs match a reference answer (Zhao et al., 2024). These neighboring uses of the term are not identical to ConsistencyCheck, but they reinforce the same methodological point: consistency is often a semantic or behavioral property that cannot be reduced to syntactic well-formedness alone.

4. Validation of LLM judges

To validate LLM judges, the paper evaluates several models as binary classifiers on the 859-item benchmark (Chen et al., 28 Oct 2025). For each item, the model must decide whether the Lean statement correctly preserves the meaning of the original problem. Performance is reported using accuracy, precision, recall, and F1 (Chen et al., 28 Oct 2025).

The reported results are as follows.

Model	Accuracy	F1
Gemini-2.5-Pro	85.8%	90.2%
Qwen3-235B-A22B	82.9%	86.5%
CriticLean-14B	79.1%	83.9%

The best reported accuracy is 85.8% from Gemini-2.5-Pro, with a corresponding F1 of 90.2%. Qwen3-235B-A22B achieves 82.9% accuracy and 86.5% F1. CriticLean-14B reaches 79.1% accuracy and 83.9% F1 (Chen et al., 28 Oct 2025).

The paper adopts Qwen3-235B-A22B as the default semantic judge for evaluation, while using CriticLean-14B in RL training for efficiency (Chen et al., 28 Oct 2025). It also argues that ReForm’s improvements are much larger than plausible evaluation noise, so the benchmark is reliable enough for the paper’s conclusions (Chen et al., 28 Oct 2025).

The results are interpreted as evidence of a substantial “classification-generation gap”: semantic judgment is itself difficult, and frontier models top out around 85.8% accuracy (Chen et al., 28 Oct 2025). This suggests that progress in judge models should not be conflated with solved evaluation.

5. Human annotation quality and the difficulty of autoformalization

One of the benchmark’s major empirical findings is that human experts themselves often introduce semantic errors in benchmark formalizations (Chen et al., 28 Oct 2025). The paper reports that 16.4% of miniF2F and 38.5% of ProofNet human-written formal statements contain semantic mistakes (Chen et al., 28 Oct 2025).

This result is one of the paper’s key messages. Autoformalization is difficult enough that even expert-curated formalizations are not guaranteed to be semantically faithful (Chen et al., 28 Oct 2025). ConsistencyCheck therefore does more than validate model judges; it also audits the quality of benchmark artifacts that might otherwise be treated as authoritative.

A plausible implication is that benchmark curation in formal mathematics requires explicit semantic review, not just expert authorship and successful compilation. The benchmark’s design embodies this view by requiring independent annotation, adjudication, and written error justifications.

The finding also addresses a common misconception: that semantic fidelity problems in autoformalization are primarily model-generated artifacts. ConsistencyCheck shows that semantic drift appears even in human-written formalizations, which reframes the problem as a property of the task itself rather than merely a weakness of current generative systems.

6. Integration with ReForm and PBSO

ConsistencyCheck is tightly connected to ReForm and PBSO (Chen et al., 28 Oct 2025). ReForm is the paper’s reflective autoformalization method: instead of a one-pass translation from natural language to Lean, the model alternates between generating a formalization and self-validating its semantic correctness, then refining based on the critique (Chen et al., 28 Oct 2025).

PBSO is designed to optimize this multi-stage sequence with heterogeneous rewards. The final formal statement receives task reward, while intermediate critiques receive auxiliary reward if they faithfully diagnose semantic errors (Chen et al., 28 Oct 2025). In this setup, the auxiliary reward is defined as

$r_{\text{aux}^t(Q,S_t,C_t) = \begin{cases} 1 & \text{if } \text{IsFaithfulCritique}(Q,S_t,C_t) \ 0 & \text{otherwise} \end{cases}$

and the bounded prospective return is

$\text{Ans}$0

before being converted into position-specific advantages for GRPO-style policy optimization (Chen et al., 28 Oct 2025).

ConsistencyCheck underpins both components. First, it supplies the semantic judge used to score correctness in training and evaluation. Second, its annotation guidelines mirror the model’s critique format, making the benchmark directly relevant to the self-validation objective (Chen et al., 28 Oct 2025). In the SFT pipeline, the authors use a prompt-based consistency check loop to create multi-turn reflective training trajectories. In RL, $\text{Ans}$1 is defined by whether a critique is faithful, again using the same consistency-checking protocol (Chen et al., 28 Oct 2025).

The benchmark is therefore not an isolated evaluation dataset. It is part of the paper’s training logic, evaluation logic, and methodological justification.

7. Benchmark implications and methodological significance

The main experimental findings around ConsistencyCheck are twofold. First, semantic judgment is itself a difficult classification problem. Second, the benchmark is robust enough to support the larger claims about ReForm, since the reported semantic improvements are far larger than the estimated judge error (Chen et al., 28 Oct 2025). In the main autoformalization results, ReForm-8B reaches 87.7% semantic consistency on miniF2F, 65.6% on ProofNet, 57.3% on Putnam, and 46.7% on AIME2025, averaging 64.3%, while ReForm-32B reaches 67.3% average semantic consistency (Chen et al., 28 Oct 2025). Under the independent CriticLean-14B evaluator, the gains remain strong, supporting the claim that the improvements are not evaluator-specific (Chen et al., 28 Oct 2025).

The paper also states several caveats relevant to benchmark design. One is that semantic consistency is inherently subtle and can be judged differently by different LLMs, which is why multiple judges are compared (Chen et al., 28 Oct 2025). Another is that benchmark quality depends on expert annotation, but expert annotations are not infallible (Chen et al., 28 Oct 2025). This implies that future autoformalization benchmarks should not treat syntactic compilation as sufficient, should include explicit semantic auditing, and should anticipate disagreement in edge cases (Chen et al., 28 Oct 2025).

In that sense, ConsistencyCheck is both a benchmark and a methodological intervention. It shows that semantic evaluation in autoformalization is hard, that LLM judges are imperfect but usable, and that expert formalizations themselves frequently contain semantic mistakes (Chen et al., 28 Oct 2025). The benchmark supports the broader thesis that autoformalization requires reflective, self-correcting methods rather than one-pass translation alone (Chen et al., 28 Oct 2025).