AriaScorer: Semantic Check for Lean Formalization

• AriaScorer is a semantic correctness module that verifies Lean formalizations are faithful to the original informal mathematics.
• It employs a multi-step process including dependency graph decomposition, Mathlib grounding, and fuzzy-integral scoring to detect semantic inconsistencies.
• By integrating term-level grounding with tools like jixia and Herald, AriaScorer improves both accuracy and precision over traditional syntactic validators.

Searching arXiv for the primary paper and adjacent works to ground citations. AriaScorer is the semantic correctness module in the Aria system, introduced in “Aria: An Agent For Retrieval and Iterative Auto-Formalization via Dependency Graph” (Wang et al., 6 Oct 2025). It is designed to determine whether a generated Lean theorem statement is not merely compilable but actually faithful to the original informal mathematics. Within Aria’s conjecture-level auto-formalization pipeline, it appears after Graph-of-Thought decomposition, Mathlib grounding, bottom-up synthesis, and compiler-guided refinement, and it defines the distinction between “compiler success rate” and “final accuracy,” where final accuracy requires both Lean compilation and AriaScorer’s semantic check (Wang et al., 6 Oct 2025).

1. Position within the Aria system

Aria is presented as a system for conjecture-level formalization in Lean that emulates human expert reasoning via a two-phase Graph-of-Thought process: first, recursively decomposing an informal statement into a dependency graph of concepts; second, constructing the formalization from grounded concepts. Known concepts are grounded in Mathlib via retrieval, missing definitions are synthesized bottom-up, and Lean compilation feedback is used to iteratively repair syntax and typing. AriaScorer is not part of this planner. It is the downstream verifier that evaluates the completed candidate statement for semantic faithfulness to the original natural-language theorem (Wang et al., 6 Oct 2025).

This placement is central to the architecture. The dependency graph structures the search, retrieval grounds concept names for synthesis, and the Lean compiler enforces well-formedness and type correctness. AriaScorer supplies the missing layer: semantic validation at the final statement level, with term-level grounding. The paper is explicit that AriaScorer does not build the dependency graph, is not described as scoring graph nodes during planning, and is not described as explicitly pruning branches of the dependency graph. Its function is narrower and more stringent: it validates whether the resulting Lean statement preserves the intended assumptions, definitions, and conclusion of the informal theorem (Wang et al., 6 Oct 2025).

2. Motivation: syntactic validity is not semantic correctness

AriaScorer is motivated by recurring failure modes in LLM-based auto-formalization. The paper emphasizes that LLMs hallucinate Lean APIs or use stale library knowledge, producing terms that look plausible but do not correspond to Mathlib. Even when code compiles, the formal statement may still be semantically mismatched: a term with a similar surface name may encode a different mathematical notion, parameter order may be wrong, or an implicit precondition may be omitted or incorrectly replaced. Prior semantic checkers and ordinary LLM judges can also be deceived by superficial textual similarity between an informal phrase and a formal term (Wang et al., 6 Oct 2025).

The resulting misconception is that compilation is a correctness certificate. Aria treats that as false. Lean compilation and type checking establish that a theorem statement is syntactically valid and type-correct, but they do not establish that it says the right thing. The paper gives the canonical failure mode as “correctly-typed but semantically wrong” translations. AriaScorer exists specifically to catch those cases. This suggests that the decisive problem in research-level auto-formalization is not only producing Lean code that type-checks, but ensuring that the generated theorem remains extensionally faithful to the original mathematics (Wang et al., 6 Oct 2025).

3. Scoring architecture and term-level grounding

AriaScorer is presented as an enhanced version of LeanScorer. Its starting point is LeanScorer’s decomposition-based evaluation: given an informal statement and a candidate Lean formalization, an LLM decomposes the original statement into atomic assumptions and conclusions, then evaluates whether the corresponding formal clauses match each subtask. Each subtask receives one of three labels: “Perfectly Match,” “Minor Inconsistency,” or “Major Inconsistency.” These judgments are aggregated via a fuzzy integral into a final score in $[0,1]$, with interpretation fixed by the paper: $0$ indicates the presence of a major error, $1$ indicates perfect alignment across all subtasks, and intermediate values reflect gradual decay from $1$ as minor inconsistencies accumulate. Acceptance is then determined by thresholding:

$$\text{accept iff } s \ge \alpha, \qquad \alpha \in [0,1].$$

The results table explicitly reports settings such as “AriaScorer($\alpha=0$)” and “AriaScorer($\alpha=0.9$)” (Wang et al., 6 Oct 2025).

Its distinctive component is a term-level semantic grounding layer inserted before subtask evaluation. AriaScorer analyzes the candidate Lean theorem with jixia, a static analyzer for Lean, which extracts every Lean term referenced in the statement. For each extracted term, AriaScorer queries the Herald informalized Mathlib dataset and retrieves “each term’s name, kind, type, value, informal name, and informal description.” This retrieved metadata is injected into the evaluation prompt together with the original informal statement, the candidate formal statement, the decomposed subtask list, and few-shot examples. The LLM therefore compares the informal theorem against a semantically annotated view of the formal theorem grounded in Mathlib documentation rather than raw Lean syntax alone (Wang et al., 6 Oct 2025).

Operationally, the checker uses six inputs: the original informal statement, the candidate Lean formalization, an LLM-produced decomposition into atomic assumptions and conclusions, the set of Lean terms extracted by jixia, the retrieved Mathlib-side metadata from Herald, and few-shot examples. The paper’s most precise prose reconstruction is: decompose the informal theorem into atomic conditions and conclusions; parse the candidate Lean theorem and extract all referenced Lean terms with jixia; retrieve authoritative Mathlib-side descriptions for these terms from Herald; build an evaluation prompt containing the informal theorem, the formal theorem, the subtask list, the retrieved term metadata, and few-shot examples; judge each subtask as a perfect match, minor inconsistency, or major inconsistency; aggregate these judgments using a fuzzy integral to a score $s \in [0,1]$; and return accept iff $s \ge \alpha$ (Wang et al., 6 Oct 2025).

Because evaluation is grounded in retrieved term semantics, AriaScorer is intended to detect discrepancies that surface-text comparison misses, including reversed parameter order, unintended type coercions, implicit preconditions encoded inside Lean definitions, divergence between a familiar mathematical notion and the actual Mathlib meaning of a term, and hallucinated explanations of Lean terms (Wang et al., 6 Oct 2025).

4. Interaction with Graph-of-Thought and the meaning of final accuracy

AriaScorer is downstream of Aria’s dependency-graph pipeline but conceptually aligned with it. Graph-of-Thought decomposition grounds concepts before synthesis; AriaScorer grounds terms again during semantic verification. This division of labor matters particularly when the synthesized theorem contains a mix of standard Mathlib terms and newly synthesized structures. The paper states that grounding evaluation in term semantics is more reliable “particularly in cases involving newly introduced or user-defined structures.” At the same time, the checker is still described as operating primarily at the final statement level rather than as a planner-in-the-loop component (Wang et al., 6 Oct 2025).

The same section of the paper clarifies several negative claims that shape the interpretation of the method. AriaScorer’s retrieval backend is not LeanSearch, which is used earlier in the GoT grounding stage; AriaScorer instead uses jixia for term extraction and Herald for semantic metadata retrieval. The indexed unit is therefore an individual Lean symbol plus its associated metadata, not a theorem-level retrieval unit. The paper also does not explicitly say that AriaScorer is used online to rerank multiple candidate formalizations or to drive self-correction loops. Its strongest documented role is post hoc semantic validation and binary acceptance. In practice, because final accuracy is defined using AriaScorer, it functions as the gate that determines whether a compiled formalization is accepted as correct (Wang et al., 6 Oct 2025).

This distinction is visible in Aria’s reported end-to-end results. On ProofNet, Aria reaches 91.6% compiler success rate and 68.5% final accuracy; on FATE-H, 89.0% compiler success and 71.0% final accuracy; on FATE-X, 69.0% compiler success and 44.0% final accuracy; and on the Conjectures dataset, 42.9% final accuracy with manual verification. The abstract further states that on FATE-X Aria achieves 44.0% final accuracy versus 24.0% for the best baseline, and on a dataset of homological conjectures Aria reaches 42.9% final accuracy while all other models score 0%. The gap between compilation and final accuracy is therefore not incidental; it is the system-level manifestation of AriaScorer’s role (Wang et al., 6 Oct 2025).

5. Validation study and characteristic strengths

The paper includes a dedicated validation study isolating semantic checking on FATE-X. The setup uses syntactically correct outputs generated by the Aria agent and asks different checkers to classify each formalization as mathematically faithful (“True”) or flawed (“False”). Ground truth labels are expert-validated by an algebra Ph.D. candidate with Mathlib contribution experience, with independent verification by a second expert. For controlled comparison, AriaScorer, LeanScorer, and Back Translation all use the same base model, Gemini-2.5-Pro, so the comparison is meant to isolate the contribution of AriaScorer’s term-level grounding (Wang et al., 6 Oct 2025).

Checker	TP / TN / FP / FN	Accuracy / Precision / Recall / F1
AriaScorer($\alpha=0$)	50 / 12 / 5 / 2	89.9% / 90.9% / 96.2% / 93.5%
AriaScorer($0$0)	42 / 15 / 2 / 10	82.6% / 95.5% / 80.8% / 87.5%
LeanScorer($0$1)	46 / 3 / 14 / 6	71.0% / 77.6% / 88.5% / 82.1%
LeanScorer($0$2)	44 / 7 / 10 / 8	73.9% / 81.5% / 84.6% / 83.0%
Back Translation	7 / 16 / 1 / 45	33.3% / 87.5% / 13.5% / 23.3%
direct Gemini	45 / 8 / 9 / 7	76.8% / 83.3% / 86.5% / 84.9%

At $0$3, AriaScorer improves over LeanScorer from 71.0% to 89.9% accuracy, reduces false positives from 14 to 5, reduces false negatives from 6 to 2, and improves F1 from 82.1% to 93.5%. At $0$4, it remains better than LeanScorer across all reported metrics except the recall tradeoff induced by stricter thresholding. The paper explicitly interprets this as evidence that term-level grounding improves both precision and recall by reducing semantic confusions (Wang et al., 6 Oct 2025).

The qualitative error analysis identifies three strengths. First, implicit semantic inclusion: in the UFD example, AriaScorer recognizes that a Lean local IsDomain instance is not an unjustified extra condition but a correct formal unpacking of the mathematical notion presupposed by UniqueFactorizationMonoid. Second, definition discrepancy detection: in the quaternion algebra example, the candidate formalization uses QuaternionAlgebra ℝ (A : ℝ) (B : ℝ) 1, whose retrieved Mathlib description does not match the intended mathematical presentation, so AriaScorer marks it as “Major Inconsistency,” whereas LeanScorer is described as being fooled by textual similarity. Third, hallucination suppression via grounding: in the Galois-group example, AriaScorer uses retrieved term semantics to distinguish QuaternionGroup 1 from the intended $0$5, noting that $0$6 corresponds to QuaternionGroup 2, and therefore marks the conclusion as a major inconsistency (Wang et al., 6 Oct 2025).

6. Thresholding, limitations, and interpretive boundaries

Threshold selection is a first-class part of the method. The paper recommends $0$7 for high-precision deployment and reports $0$8 to show intrinsic detection power. The validation table makes the tradeoff explicit: $0$9 yields the best recall and overall F1, whereas $1$0 yields the highest precision. The authors therefore use $1$1 for the rest of their experiments as a practical high-precision operating point, accepting lower recall (Wang et al., 6 Oct 2025).

Several limitations are also explicit. AriaScorer depends on an accurate curated mapping from Lean terms to informal descriptions, here provided by the Herald informalized Mathlib dataset, so its grounding is strongest where Mathlib coverage and annotation quality are good. Although the paper states that AriaScorer handles “newly introduced or user-defined structures” better than purely textual methods, the mechanism appears to rely on visible local definitions and surrounding retrieved terms; the paper does not claim full semantic understanding of arbitrary novel definitions beyond what can be inferred from context. Implementation details remain partly underspecified: there is no dedicated pseudocode block, no exact fuzzy-integral formula, no exact prompt template, and no specified token budget, truncation policy, top-$1$2 retrieval count, or indexing architecture beyond the described contents of the prompt (Wang et al., 6 Oct 2025).

The resulting interpretive boundary is precise. AriaScorer is neither a replacement for the Lean compiler nor a generic LLM judge that reasons from surface resemblance. It is a retrieval-augmented semantic rejection stage whose central innovation is term-level grounding in authoritative Mathlib-side meanings. Its role in Aria is therefore best understood as semantic gatekeeping: GoT and compiler-in-the-loop synthesis produce candidate Lean statements, and AriaScorer decides whether those statements actually mean the same thing as the original mathematics (Wang et al., 6 Oct 2025).