---
title: 'COVCAL: Risk-Controlled Selective Prediction'
url: https://www.emergentmind.com/topics/covcal
type: topic
---

# COVCAL: Risk-Controlled Selective Prediction

Searching arXiv for the exact topic and closely related uses of “COVCAL” / “CovCal”.
COVCAL, introduced as **CovCal** in “Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning,” is a **risk-controlled selective prediction wrapper** for using Lean as a judge of natural-language mathematical answers. Its defining premise is that Lean provides a **partial observation** of correctness: a kernel-checked proof is strong positive evidence for the generated Lean statement, but proof failure is ambiguous because it may reflect failed autoformalization, typechecking, library retrieval, or proof search rather than a wrong answer. COVCAL therefore treats Lean-as-judge as a **partial-observation selective prediction problem**, not as a binary oracle, and returns either a formally selected answer or abstention, with a finite-sample certificate on the selective risk of accepted answers [2605.28365].

## 1. Motivation and conceptual framing

COVCAL was proposed for a setting in which a language model generates natural-language mathematical answers, an autoformalization pipeline translates candidate answers into Lean statements, and Lean supplies trace diagnostics such as whether a statement was proved, typechecked, illtyped, timed out, or remained unformalized. The central empirical motivation is that raw Lean success or failure is not, by itself, an adequate answer-selection rule. Proof success is asymmetrically informative, while proof failure is heterogeneous and weakly informative.

The paper’s main empirical finding is a pronounced **coverage cliff**. The proof-winning answer is correct **96%** of the time at high proved coverage but only **20%** at low proved coverage. Table-level summaries in the paper make this concrete: for \(C_{prf}<0.25\), winner accuracy is **0.20**; for \(C_{prf}\ge 0.75\), it is **0.96**. The same effect is sharper for formal margin: when \(M<0\), winner accuracy is **0.10**; when \(M\ge 0.5\), it is **0.98**. This establishes that a proved answer is trustworthy mainly when it dominates the formally visible candidate mass and no heavy unresolved rival remains [2605.28365].

The signal is also sparse. On the main MATH-500 run, Lean proves at least one answer class for only **28%** of problems using the 7B autoformalizer. Even proof success is not equivalent to faithful verification of the original problem. The paper reports that an automated audit labels **73.6%** of proved artifacts as non-trivial and correct, but a manual audit finds only about **\(\approx 43\%\)** of proved statements are actually **faithful** to the original problem. This is why COVCAL is organized around **selective acceptance**: trust the formal signal only when the observable Lean trace falls into a region whose risk can be certified; otherwise abstain.

## 2. Formal setup and observational statistics

The formal object operated on by COVCAL is not raw text and not Lean proof status alone, but a structured record built from sampled answers, normalized answer classes, class weights, and class-level formal diagnostics. For a problem \(x\), a generator produces \(K\) candidate answers \(a_1,\dots,a_K\) with nonnegative weights \(q_j(x)\) summing to one. Candidates are grouped into normalized answer classes \(e_x(a_j)\in C(x)\), with class weights
\[
Q_c(x)=\sum_{j:e_x(a_j)=c} q_j(x),\qquad \sum_{c\in C(x)}Q_c(x)=1.
\]

For each candidate \(a_j\), the formal pipeline attempts autoformalization and verification and records a status
\[
s_j \in S,\qquad S=\{\mathtt{proved},\mathtt{typechecked},\mathtt{illtyped},\mathtt{timeout},\mathtt{unformalized}\}.
\]
At class level, the paper defines
\[
P_c(x)=I\{\exists j:e_x(a_j)=c,\ s_j=\mathtt{proved}\},
\]
and
\[
T_c(x)=I\{\exists j:e_x(a_j)=c,\ s_j\in S_{typ}\},\qquad S_{typ}=\{\mathtt{proved},\mathtt{typechecked},\mathtt{timeout}\}.
\]
Thus \(P_c(x)=1\) means class \(c\) has at least one kernel-checked proof, while \(T_c(x)=1\) means the class at least reached a well-formed elaborated statement.

COVCAL’s base formal selector is
\[
g_{\mathrm F}(x)=c^+(x),
\]
where \(c^+(x)\) is the highest-weight proved class. If no class is proved, or if there is conflict among inequivalent proved classes, then
\[
g_{\mathrm F}(x)=\bot.
\]
The diagnostic statistics then quantify how much of the answer mass is formally visible and whether the proved class dominates unresolved rivals. Typed coverage is
\[
C_{typ}(x)=\sum_{c\in C(x)} Q_c(x)T_c(x),
\]
proved coverage is
\[
C_{prf}(x)=\sum_{c\in C(x)} Q_c(x)P_c(x),
\]
unresolved rival mass is
\[
R_{unres}(x)=\max_{c\ne c^+(x):P_c(x)=0} Q_c(x),
\]
with \(R_{unres}(x)=0\) if every rival is also proved or no rival exists, and formal margin is
\[
M(x)=Q_{c^+(x)}(x)-R_{unres}(x),
\]
with \(M(x)=-\infty\) if no class is proved. The formal margin is conceptually central because it measures whether the best proved class actually dominates the strongest unresolved rival.

## 3. Acceptance rule and operational workflow

COVCAL accepts only when the formal trace clears calibrated thresholds on visibility and dominance. For threshold tuple
\[
\tau=(\tau_{typ},\tau_{prf},\tau_M),
\]
the acceptance rule is
\[
A_\tau(x)=I\{C_{typ}(x)\ge\tau_{typ},\ C_{prf}(x)\ge\tau_{prf},\ M(x)\ge\tau_M,\ g_{\mathrm F}(x)\ne\bot,\ \text{no conflict}\}.
\]
If \(A_\tau(x)=1\), COVCAL returns \(g_{\mathrm F}(x)\); otherwise it abstains.

The practical pipeline is deliberately modular. For each problem, the system samples \(K=32\) natural-language solutions using **Qwen2.5-Math-7B-Instruct**, normalizes them into answer classes, computes self-consistency weights \(Q_c(x)\), formalizes the top four answer classes into Lean, generates several artifacts per class with one repair pass, runs Lean 4.21.0 + Mathlib with a small tactic portfolio and **20s/script**, aggregates artifact statuses to class indicators \(P_c\) and \(T_c\), computes \(C_{typ}\), \(C_{prf}\), \(R_{unres}\), and \(M\), applies \(g_{\mathrm F}\), and finally applies a calibrated threshold rule [2605.28365].

The threshold grid is fixed in advance:
\[
T_{typ}=\{0,0.25,0.5,0.75,0.9\},\qquad
T_{prf}=\{0,0.1,0.25,0.5,0.75\},
\]
\[
T_M=\{-0.5,0,0.1,0.25,0.5\},\qquad
T=T_{typ}\times T_{prf}\times T_M,
\]
with \(|T|=125\). Although the formal definition uses three thresholds, the paper reports that the strongest single signal is the **formal margin**: margin-only matches joint COVCAL almost exactly on the main run, with accepted accuracy **0.932** and accepted fraction about **0.209** in both cases.

## 4. Finite-sample risk control

The method’s statistical core is a finite-sample certificate on the **population selective risk**
\[
R(\tau)=\Pr[g_{\mathrm F}(X)\ne C^\star\mid A_\tau(X)=1],
\]
with the convention \(R(\tau)=0\) if \(\Pr[A_\tau(X)=1]=0\). On calibration data
\[
D_{\mathrm{cal}}=\{(X_i,C_i^\star)\}_{i=1}^{n},
\]
the accepted count and accepted-error count are
\[
m_\tau=\sum_{i=1}^n A_\tau(X_i),\qquad
k_\tau=\sum_{i=1}^n A_\tau(X_i)I\{g_{\mathrm F}(X_i)\ne C_i^\star\}.
\]
The user specifies a target selective-risk level \(\epsilon\) and failure probability \(\delta\); in the main experiments, \(\epsilon=0.15\) and \(\delta=0.05\).

COVCAL uses a one-sided **Clopper--Pearson upper bound** \(U_\alpha(k,m)\) on a binomial error rate. For fixed \(\tau\), accepted calibration examples are i.i.d. from the conditional distribution given acceptance, so
\[
k_\tau \mid m_\tau \sim \mathrm{Binomial}(m_\tau,R(\tau)).
\]
This yields two certification regimes. In the **Bonferroni regime**, the calibration split is used both to search over all \(125\) threshold cells and to certify them, giving the simultaneous guarantee
\[
R(\tau)\le U_{\delta/|T|}(k_\tau,m_\tau)\qquad \forall \tau\in T
\]
with probability at least \(1-\delta\). The selected threshold is
\[
\hat\tau_{\mathrm{Bonf}}=\arg\max_{\tau\in T} m_\tau
\quad\text{s.t.}\quad
U_{\delta/|T|}(k_\tau,m_\tau)\le \epsilon.
\]

In the **dev-then-cal regime**, threshold selection is performed on an independent development split and only one threshold is certified on calibration. This removes the union-bound penalty and uses level \(\delta\), not \(\delta/|T|\). The paper’s stated guarantee is that, with probability at least \(1-\delta\), if the output threshold is not \(\bot\), then its population selective risk satisfies \(R(\widehat\tau_{\mathrm{out}})\le U_\delta(k,m)\le \epsilon\) [2605.28365].

The certificate is intentionally narrow. It controls the accuracy of **accepted formal predictions** only. It does not control overall system accuracy including fallback, correctness of abstained examples, mathematical truth of unresolved classes, or faithfulness of the Lean statement beyond what is reflected in calibration labels.

## 5. Empirical performance and dependence on formalization coverage

The main experiment uses a filtered short-answer subset of **MATH-500**, retaining **378 of 500** examples after excluding proof-only, diagram-dependent, and non-normalizable items. The split is **76 dev**, **151 calibration**, and **151 test**. The paper also reports **20 bootstrap dev/cal/test partitions** over fixed logs, so variability reflects split choice rather than rerunning generation or autoformalization.

The status distribution in the main run was: **unformalized: 741 (52.8%)**, **typechecked: 271 (19.3%)**, **illtyped: 251 (17.9%)**, **proved: 140 (10.0%)**, and **timeout: 0**. This shows that the main bottleneck is before proof search rather than timeout on well-formed statements. With the 7B autoformalizer, proved coverage reaches only **28%** of problems. Under that regime, **Bonferroni rejects all \(20/20\) bootstrap partitions**, while **dev-then-cal certifies \(12/20\)**.

The paper’s central feasibility result is that stronger autoformalization changes the certification problem qualitatively. Replacing the formalizer with **Goedel-Prover-V2-8B** raises proved coverage from **28%** to **79%** at similar proof-winner precision. This “feasibility flip” yields **Bonferroni feasible on \(17/20\)** bootstrap partitions and **dev-then-cal feasible on \(20/20\)**. Under this stronger formalizer, the method accepts about **\(\approx 48\%\)** of problems at **0.98 accepted accuracy** [2605.28365].

On the main run, the comparison to simpler baselines is explicit. **Self-consistency** has overall accuracy **0.910**. **Proof-existence abstention** has accepted accuracy **0.877** at accepted fraction **0.220**. **COVCAL (dev-then-cal)** has accepted accuracy **0.932** at accepted fraction **0.209**. **COVCAL + fallback** has overall accuracy **0.905**. The paper is explicit that, because self-consistency is already **91%** accurate, the contribution is **not** higher raw end-to-end accuracy; it is a finite-sample certificate identifying when partial formal evidence can be trusted.

## 6. Interpretation, misconceptions, and scope

A persistent misconception addressed by COVCAL is that “Lean proved something” means the natural-language answer has been verified. The paper rejects that equivalence. Lean proof success is a strong positive signal for the generated formal statement, but the generated statement may be semantically unfaithful to the original problem. Conversely, lack of proof is not evidence of falsity in any simple sense; it may reflect autoformalization failure or proof-search limitations. This asymmetry is the reason COVCAL is a **selective predictor** with abstention rather than a direct correctness oracle.

A second misconception is that any proved class should be selected whenever one exists. The paper’s proposition on unresolved answer classes states, in substance, that if a verifier-only selector picks a proved class while an unresolved inequivalent rival remains with positive weight and no formal evidence deciding it, then there exist two latent correctness assignments compatible with the same formal observation that disagree on whether the selected class is correct. A plausible implication is that low coverage and heavy unresolved rivals make pointwise certification impossible without abstention.

The method’s limitations are also explicit. Its certificate scope is restricted to accepted formal predictions. It assumes calibration/deployment exchangeability, and dev-then-cal additionally requires strict dev/cal independence. Its usefulness is governed by autoformalizer quality: with weak formalization coverage, the method becomes uninformative and may return reject-all. Faithfulness remains imperfect even under proof success, and the empirical evidence is concentrated on MATH-500 and one candidate-generation pipeline. The reported bootstrap variability is over splits, not over fresh generation or formalization randomness.

Within those limits, COVCAL occupies a specific place in the methodology of machine-checked evaluation. It is neither a theorem prover nor a generic calibration method for all prediction systems. It is a finite-sample, Lean-trace-based **risk-controlled selective wrapper** for deciding **when Lean-as-judge is trustworthy** and when it is out of coverage.

Source: https://www.emergentmind.com/topics/covcal