The Signal-Coverage Matrix: Stratifying Type and Semantic Errors in Statement Autoformalization

Abstract: Headline type-correctness (TC\%) of LLM autoformalization has climbed from $\sim$53\% to $\sim$76\% in two years, yet this scalar conceals which errors each method resolves. We propose a signal-coverage matrix that crosses the Lean elaborator (pass/fail) with a semantic-equivalence judgment (equivalent/not), sorting every output into one of four cells: true success (TS), type-only (TO), semantic-only (SO), or both fail (BF). On ProofNet# and MiniF2F-test with DeepSeek V4-Pro across Vanilla, Lean-Retry, Sample-Filter, and Stratified Autoformalization (SAF): (1) the +34 to +36 TS gain across the three elab-feedback methods is $\sim$64\% type-stratum recovery, with SO flat on net (87.5\% of original semantic errors rescued, 8 newly created). (2) The TO-to-TS rate is 23/61 for each method (Wilson 95\% CI [26.6\%, 50.3\%]), and this stratum-level recovery rate predicts $Δ$TS on held-out methods to within 2/186 and renders $Δ$TC linear in the Vanilla elab-fail rate across six (model, dataset) cells ($R<sup>2=0.96$). (3) The two judges disagree by 26 to 37 pp on elab-feedback outputs (vs. 7 pp on Vanilla), with 30 to 56\% of symbolic-judge false negatives traceable to elaborator-forced rewrites. The persistent residual reduces to two gold-formalization errors. TC\% gains should be credited by which cell moved, not by the scalar alone.

• The paper introduces a signal-coverage matrix that decomposes autoformalization errors into type (syntactic) and semantic strata using a 2×2 diagnostic framework.
• It employs elaborator-feedback methods and dual-judge evaluation (LLM and symbolic) across benchmarks to precisely measure true successes and error recoveries.
• Results reveal robust recovery of type-only errors but persistent semantic errors, highlighting the need for enhanced feedback mechanisms in autoformalization.

The Signal-Coverage Matrix for Autoformalization: Stratifying Type and Semantic Errors

Introduction

The paper "The Signal-Coverage Matrix: Stratifying Type and Semantic Errors in Statement Autoformalization" (2606.28013) presents a detailed analysis of error modalities in LLM-driven statement autoformalization, augmenting the typical reporting of type-correctness (TC%) and semantic-faithfulness (SF%) with a diagnostic $2{\times}2$ error matrix. By dissecting failures as either type-stratum (syntactic, not type-correct in Lean) or semantic-stratum (type-correct but denotationally incorrect), and examining how various algorithms affect these strata, the paper provides actionable insights into the limitations and current ceilings of LLM-based autoformalization and the adequacy of both symbolic and LLM-based semantic judges.

Diagnostic Framework: The Signal-Coverage Matrix

At the core of the paper is the signal-coverage matrix, which crosses the Lean elaborator signal (pass/fail) with a semantic-equivalence judge (equivalent/not), assigning each output to one of four categories:

• True Success (TS): type-correct and semantically faithful
• Type-Only Failure (TO): semantically faithful, syntactically invalid
• Semantic-Only Failure (SO): type-correct but semantically invalid
• Both Fail (BF): neither type-correct nor semantically faithful

This stratification enables precise localization of aggregate metric gains and exposes previously conflated error modes.

Figure 1: The $2\times 2$ signal-coverage matrix illustrating the joint action of the Lean elaborator and semantic equivalence judge.

The matrix allows the decomposition of headline TC% and SF% by explicitly identifying which failures are addressed by given methods, and which persist. Notably, SO and TO correspond to the off-diagonal cells where type-system and semantic feedback disagree.

Empirical Evaluation: Methods, Judges, and Datasets

The study evaluates Vanilla, Lean-Retry, Sample-Filter, and the paper's Stratified Autoformalization (SAF) methods on DeepSeek V4-Pro, Qwen3.5-Plus, and MiMo-v2.5-Pro across ProofNet$^{\#}$ (type-complex) and MiniF2F (type-simple) benchmarks.

• Judging Protocol: Semantic faithfulness is assessed via both an LLM judge (Claude Opus 4.7) and a symbolic GTED judge (generalized tree edit distance at $\tau=0.5$). This dual protocol supports direct analysis of judge agreement and, crucially, exposes the impact of surface-level type-checking induced rewrites.

Figure 2: The Vanilla signal-coverage matrix (DeepSeek V4-Pro × ProofNet$^{\#}$), highlighting off-diagonal cells where automatic feedback and the LLM semantic judge disagree.

Figure 3: TC\% and SF\% progression across Vanilla and elaborator-feedback methods, showing improvement in true successes.

Key Numerical Results

The major findings are as follows:

• All three elaborator-feedback methods (Lean-Retry, Sample-Filter, SAF) exhibit nearly identical performance, delivering a +34 to +36 problem gain in true successes (TS) on ProofNet$^{\#}$, corresponding to $\sim$64% recovery of type-only failures (TO).
• The TO→TS recovery rate is $23/61$ (Wilson 95% CI [26.6%, 50.3%]), identical across methods despite different architectures and LLM call topologies, suggesting that the elaborator signal is exhausted with the given refinement budget.
• The semantic-only failure (SO) rate is net-flat across methods, with 87.5% of SO errors rescued but 8 new ones created per method, resulting in high error turnover but no aggregate semantic improvement.
• Cross-method calibration using per-stratum recovery rates accurately predicts $\Delta$TS for other methods to within $2/186$ problems; $2\times 2$0TC fits a linear regression in the Vanilla elaborator-fail rate across models and datasets $2\times 2$1, emphasizing the systematized nature of improvement.
• On MiniF2F (type-simple), the SO rate is nearly unchanged relative to ProofNet$2\times 2$2, but TO rates drop significantly, confirming the diagnostic relevance of the type-stratum for more challenging datasets.

Figure 4: Per-problem Vanilla to Lean-Retry transitions, with most TS gains attributed to the TO→TS stratum.

Figure 5: Cross-model evaluation on ProofNet$2\times 2$3 and MiniF2F shows that TC gains are robust across architectures and datasets.

Surface-Form Effects and Judge Disagreement

A pronounced discrepancy arises between LLM and symbolic judges on elaborator-feedback outputs: Opus-based SF% rises by $2\times 2$4–$2\times 2$5 points compared to GTED. The underlying mechanism is that surface rewrites forced by the elaborator (e.g., changing function type expressions, quantifier placements) yield operator trees that differ structurally but denote the same proposition, which LLM judges correctly deem equivalent while GTED does not.

• SAF, by rigidly fixing Lean output as a deterministic function of the IR, isolates this phenomenon: its Opus-vs-GTED gap reaches $2\times 2$6pp.
• Error taxonomy indicates that structural rewrites, not merely notational changes, dominate the disagreement.

The implication is that single-judge symbolic SF% under-credits elaborator-feedback methods by $2\times 2$7pp—an important consideration for benchmarking and future method evaluation.

K-Saturation and Refinement Limits

Figure 6: SAF K-saturation curve demonstrates that refinement gains saturate quickly, with all feedback methods converging to nearly identical ceilings by $2\times 2$8.

• Most recoverable TO errors are resolved within the first refinement iteration ($2\times 2$9), with diminishing returns at higher $^{\#}$0.
• A fixed, irreducible subset of TO errors remains across all methods, typically those where the LLM produces plausible but non-existent Mathlib API names—cases the current feedback paradigm cannot rectify.

Practical and Theoretical Implications

Benchmarking and Evaluation Protocols

The per-stratum decomposition reveals that aggregate TC% and SF% are insufficient for evaluating autoformalization progress; reporting must include TS/SO/TO/BF counts to expose the nature and locus of improvements. The strong judge disagreement on semantic faithfulness for elab-feedback methods makes clear that an independently calibrated LLM judge, or ideally an expert-plus-symbolic alliance, is required for accurate assessment.

Ceiling Effects and Limitations of Current Feedback

With all methods converging in both TO→TS rescue rates and overall TC%, future progress in autoformalization requires new feedback signals that directly address semantic stratum failures—not just type elaboration. The durable TO and SO error pools identified suggest explicit programmatic and semantic guidance, beyond surface-form refinement, as a necessary next axis of development.

Gold Data Quality

The identification of durable SO cases as gold-formalization errors underlines the need for carefully audited benchmarks, especially when SO error rates approach the sub-percent range, highlighting a limit of scale alone for further advances.

Future Directions

• Development of stronger semantic feedback channels: Differentiating between surface-level and deep semantic errors requires more than elaborator guidance. Future work should target feedback mechanisms sensitive to deeper denotational equivalence.
• Canonicalization-aware metrics: To bridge the Opus-GTED gap, metrics that perform operator-tree matching under normalization (e.g., whnf reduction) or enriched equivalence classes could be introduced.
• Tool augmentation: Explicit mapping between NL statements and the correct Mathlib APIs, possibly through retrieval-based assistance, could help close the durable TO gap.
• Robust evaluation pipelines: Incorporating both LLM-based and symbolic judges, with clear calibration and auditability, becomes mandatory for fair method comparison.

Conclusion

The signal-coverage matrix and per-problem transition analysis elucidate the fundamental mechanisms by which contemporary autoformalization methods make and fail to make progress. Strong numerical regularities (e.g., uniform TO→TS rates and the strict alignment between aggregate improvements and error stratum dynamics) clarify where current approaches plateau. Judge divergence underlines the necessity of semantic-aware, calibration-robust evaluation. Overall, progress will hinge on feedback modalities capable of diagnosing and guiding repair of not just the type, but the semantic, content of formalization outputs.

For complete references, see (2606.28013).