COVCAL: Risk-Controlled Selective Prediction

• The paper introduces COVCAL, a risk-controlled selective prediction wrapper that uses Lean’s partial observations to certify mathematical answers.
• It employs a modular pipeline combining autoformalization, formal trace diagnostics, and calibrated thresholds to manage uncertainty in answer verification.
• Empirical results reveal a pronounced coverage cliff, with high proved coverage achieving 96% accuracy compared to only 20% at low coverage.

Searching arXiv for the exact topic and closely related uses of “COVCAL” / “CovCal”. COVCAL, introduced as CovCal in “Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning,” is a risk-controlled selective prediction wrapper for using Lean as a judge of natural-language mathematical answers. Its defining premise is that Lean provides a partial observation of correctness: a kernel-checked proof is strong positive evidence for the generated Lean statement, but proof failure is ambiguous because it may reflect failed autoformalization, typechecking, library retrieval, or proof search rather than a wrong answer. COVCAL therefore treats Lean-as-judge as a partial-observation selective prediction problem, not as a binary oracle, and returns either a formally selected answer or abstention, with a finite-sample certificate on the selective risk of accepted answers (Bourigault et al., 27 May 2026).

1. Motivation and conceptual framing

COVCAL was proposed for a setting in which a LLM generates natural-language mathematical answers, an autoformalization pipeline translates candidate answers into Lean statements, and Lean supplies trace diagnostics such as whether a statement was proved, typechecked, illtyped, timed out, or remained unformalized. The central empirical motivation is that raw Lean success or failure is not, by itself, an adequate answer-selection rule. Proof success is asymmetrically informative, while proof failure is heterogeneous and weakly informative.

The paper’s main empirical finding is a pronounced coverage cliff. The proof-winning answer is correct 96% of the time at high proved coverage but only 20% at low proved coverage. Table-level summaries in the paper make this concrete: for $C_{prf}<0.25$, winner accuracy is 0.20; for $C_{prf}\ge 0.75$, it is 0.96. The same effect is sharper for formal margin: when $M<0$, winner accuracy is 0.10; when $M\ge 0.5$, it is 0.98. This establishes that a proved answer is trustworthy mainly when it dominates the formally visible candidate mass and no heavy unresolved rival remains (Bourigault et al., 27 May 2026).

The signal is also sparse. On the main MATH-500 run, Lean proves at least one answer class for only 28% of problems using the 7B autoformalizer. Even proof success is not equivalent to faithful verification of the original problem. The paper reports that an automated audit labels 73.6% of proved artifacts as non-trivial and correct, but a manual audit finds only about $\approx 43\%$ of proved statements are actually faithful to the original problem. This is why COVCAL is organized around selective acceptance: trust the formal signal only when the observable Lean trace falls into a region whose risk can be certified; otherwise abstain.

2. Formal setup and observational statistics

The formal object operated on by COVCAL is not raw text and not Lean proof status alone, but a structured record built from sampled answers, normalized answer classes, class weights, and class-level formal diagnostics. For a problem $x$, a generator produces $K$ candidate answers $a_1,\dots,a_K$ with nonnegative weights $q_j(x)$ summing to one. Candidates are grouped into normalized answer classes $e_x(a_j)\in C(x)$, with class weights

$C_{prf}\ge 0.75$0

For each candidate $C_{prf}\ge 0.75$1, the formal pipeline attempts autoformalization and verification and records a status

$C_{prf}\ge 0.75$2

At class level, the paper defines

$C_{prf}\ge 0.75$3

and

$C_{prf}\ge 0.75$4

Thus $C_{prf}\ge 0.75$5 means class $C_{prf}\ge 0.75$6 has at least one kernel-checked proof, while $C_{prf}\ge 0.75$7 means the class at least reached a well-formed elaborated statement.

COVCAL’s base formal selector is

$C_{prf}\ge 0.75$8

where $C_{prf}\ge 0.75$9 is the highest-weight proved class. If no class is proved, or if there is conflict among inequivalent proved classes, then

$M<0$0

The diagnostic statistics then quantify how much of the answer mass is formally visible and whether the proved class dominates unresolved rivals. Typed coverage is

$M<0$1

proved coverage is

$M<0$2

unresolved rival mass is

$M<0$3

with $M<0$4 if every rival is also proved or no rival exists, and formal margin is

$M<0$5

with $M<0$6 if no class is proved. The formal margin is conceptually central because it measures whether the best proved class actually dominates the strongest unresolved rival.

3. Acceptance rule and operational workflow

COVCAL accepts only when the formal trace clears calibrated thresholds on visibility and dominance. For threshold tuple

$M<0$7

the acceptance rule is

$M<0$8

If $M<0$9, COVCAL returns $M\ge 0.5$0; otherwise it abstains.

The practical pipeline is deliberately modular. For each problem, the system samples $M\ge 0.5$1 natural-language solutions using Qwen2.5-Math-7B-Instruct, normalizes them into answer classes, computes self-consistency weights $M\ge 0.5$2, formalizes the top four answer classes into Lean, generates several artifacts per class with one repair pass, runs Lean 4.21.0 + Mathlib with a small tactic portfolio and 20s/script, aggregates artifact statuses to class indicators $M\ge 0.5$3 and $M\ge 0.5$4, computes $M\ge 0.5$5, $M\ge 0.5$6, $M\ge 0.5$7, and $M\ge 0.5$8, applies $M\ge 0.5$9, and finally applies a calibrated threshold rule (Bourigault et al., 27 May 2026).

The threshold grid is fixed in advance: $\approx 43\%$0

$\approx 43\%$1

with $\approx 43\%$2. Although the formal definition uses three thresholds, the paper reports that the strongest single signal is the formal margin: margin-only matches joint COVCAL almost exactly on the main run, with accepted accuracy 0.932 and accepted fraction about 0.209 in both cases.

4. Finite-sample risk control

The method’s statistical core is a finite-sample certificate on the population selective risk

$\approx 43\%$3

with the convention $\approx 43\%$4 if $\approx 43\%$5. On calibration data

$\approx 43\%$6

the accepted count and accepted-error count are

$\approx 43\%$7

The user specifies a target selective-risk level $\approx 43\%$8 and failure probability $\approx 43\%$9; in the main experiments, $x$0 and $x$1.

COVCAL uses a one-sided Clopper--Pearson upper bound $x$2 on a binomial error rate. For fixed $x$3, accepted calibration examples are i.i.d. from the conditional distribution given acceptance, so

$x$4

This yields two certification regimes. In the Bonferroni regime, the calibration split is used both to search over all $x$5 threshold cells and to certify them, giving the simultaneous guarantee

$x$6

with probability at least $x$7. The selected threshold is

$x$8

In the dev-then-cal regime, threshold selection is performed on an independent development split and only one threshold is certified on calibration. This removes the union-bound penalty and uses level $x$9, not $K$0. The paper’s stated guarantee is that, with probability at least $K$1, if the output threshold is not $K$2, then its population selective risk satisfies $K$3 (Bourigault et al., 27 May 2026).

The certificate is intentionally narrow. It controls the accuracy of accepted formal predictions only. It does not control overall system accuracy including fallback, correctness of abstained examples, mathematical truth of unresolved classes, or faithfulness of the Lean statement beyond what is reflected in calibration labels.

5. Empirical performance and dependence on formalization coverage

The main experiment uses a filtered short-answer subset of MATH-500, retaining 378 of 500 examples after excluding proof-only, diagram-dependent, and non-normalizable items. The split is 76 dev, 151 calibration, and 151 test. The paper also reports 20 bootstrap dev/cal/test partitions over fixed logs, so variability reflects split choice rather than rerunning generation or autoformalization.

The status distribution in the main run was: unformalized: 741 (52.8%), typechecked: 271 (19.3%), illtyped: 251 (17.9%), proved: 140 (10.0%), and timeout: 0. This shows that the main bottleneck is before proof search rather than timeout on well-formed statements. With the 7B autoformalizer, proved coverage reaches only 28% of problems. Under that regime, Bonferroni rejects all $K$4 bootstrap partitions, while dev-then-cal certifies $K$5.

The paper’s central feasibility result is that stronger autoformalization changes the certification problem qualitatively. Replacing the formalizer with Goedel-Prover-V2-8B raises proved coverage from 28% to 79% at similar proof-winner precision. This “feasibility flip” yields Bonferroni feasible on $K$6 bootstrap partitions and dev-then-cal feasible on $K$7. Under this stronger formalizer, the method accepts about $K$8 of problems at 0.98 accepted accuracy (Bourigault et al., 27 May 2026).

On the main run, the comparison to simpler baselines is explicit. Self-consistency has overall accuracy 0.910. Proof-existence abstention has accepted accuracy 0.877 at accepted fraction 0.220. COVCAL (dev-then-cal) has accepted accuracy 0.932 at accepted fraction 0.209. COVCAL + fallback has overall accuracy 0.905. The paper is explicit that, because self-consistency is already 91% accurate, the contribution is not higher raw end-to-end accuracy; it is a finite-sample certificate identifying when partial formal evidence can be trusted.

6. Interpretation, misconceptions, and scope

A persistent misconception addressed by COVCAL is that “Lean proved something” means the natural-language answer has been verified. The paper rejects that equivalence. Lean proof success is a strong positive signal for the generated formal statement, but the generated statement may be semantically unfaithful to the original problem. Conversely, lack of proof is not evidence of falsity in any simple sense; it may reflect autoformalization failure or proof-search limitations. This asymmetry is the reason COVCAL is a selective predictor with abstention rather than a direct correctness oracle.

A second misconception is that any proved class should be selected whenever one exists. The paper’s proposition on unresolved answer classes states, in substance, that if a verifier-only selector picks a proved class while an unresolved inequivalent rival remains with positive weight and no formal evidence deciding it, then there exist two latent correctness assignments compatible with the same formal observation that disagree on whether the selected class is correct. A plausible implication is that low coverage and heavy unresolved rivals make pointwise certification impossible without abstention.

The method’s limitations are also explicit. Its certificate scope is restricted to accepted formal predictions. It assumes calibration/deployment exchangeability, and dev-then-cal additionally requires strict dev/cal independence. Its usefulness is governed by autoformalizer quality: with weak formalization coverage, the method becomes uninformative and may return reject-all. Faithfulness remains imperfect even under proof success, and the empirical evidence is concentrated on MATH-500 and one candidate-generation pipeline. The reported bootstrap variability is over splits, not over fresh generation or formalization randomness.

Within those limits, COVCAL occupies a specific place in the methodology of machine-checked evaluation. It is neither a theorem prover nor a generic calibration method for all prediction systems. It is a finite-sample, Lean-trace-based risk-controlled selective wrapper for deciding when Lean-as-judge is trustworthy and when it is out of coverage.