Proper Calibration Errors: Theory and Applications

• Proper calibration errors are discrepancies defined relative to a normative criterion, indicating how well model predictions match the true conditional outcome distributions.
• They are quantified using proper scoring rules and divergences like KL and L2, with estimation methods including variational formulations and kernel-based techniques.
• Applications span recalibration in multiclass predictions, gravitational-wave detector calibration, cosmic shear adjustments, and survey photometric calibrations.

Proper calibration errors designate calibration discrepancies defined relative to a normative criterion. In probabilistic prediction, they are calibration errors induced by proper scoring rules, typically expressed as expectations of divergences between a model prediction and the conditional distribution of outcomes given that prediction. In several measurement sciences, the same phrase is used for calibration inaccuracies that are small enough that inference remains limited by random noise rather than by calibration systematics (Popordanoska et al., 2023, Berta et al., 27 Feb 2026, Hall et al., 2017).

1. Proper-scoring-rule foundations

In multiclass probabilistic prediction, a classifier is a map $f:\mathcal{X}\to\Delta_k$, with $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$, and the conditional class distribution given the prediction is

$$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$

Perfect calibration is the requirement

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$

Given a divergence $d:\Delta_k\times\Delta_k\to[0,\infty)$, the calibration error is

$\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$

which vanishes iff $f(X)=C$ almost surely (Berta et al., 27 Feb 2026).

A proper loss $\ell(p,y)$ is one for which the expected loss

$L_\ell(p;q):=\mathbb{E}_{Y\sim q}[\ell(p,Y)]$

is minimized at $p=q$, and it is strictly proper when that minimizer is unique. For proper losses characterized on the simplex by a concave function $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$0, the induced divergence is

$\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$1

and Bröcker’s decomposition gives

$\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$2

The first term is exactly a proper calibration error: $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$3 This formulation makes calibration the excess risk attributable to miscalibration under a proper scoring rule (Berta et al., 27 Feb 2026).

An equivalent Bregman-divergence formulation is

$\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$4

where $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$5 is the negative Bayes risk associated with a differentiable proper score. For $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$6, this yields the squared canonical $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$7 calibration error; for $\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$8, it yields a Kullback–Leibler calibration error induced by cross-entropy (Popordanoska et al., 2023). In a closely related formulation, strong calibration is written as

$\Delta_k=\{p\in[0,1]^k\mid \sum_i p_i=1\}$9

and proper calibration errors are those that vanish exactly under this stronger distributional condition rather than only under top-label confidence matching (Gruber et al., 2022).

2. Estimation theory and variational formulations

The classical empirical route estimates calibration error by binning predictions, computing bin-wise average confidence and empirical frequency, and summing weighted discrepancies. In the binary case, Expected Calibration Error is typically approximated by

$$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$0

This approach is biased, inconsistent, sensitive to bin choice, and particularly problematic in multiclass settings because binning the simplex suffers from the curse of dimensionality (Berta et al., 27 Feb 2026). More generally, empirical studies of ECE estimators show substantial dependence on the number of bins, on adaptive versus uniform binning, and on sample size; reliable estimation typically requires several hundred validation points rather than very small holdout sets (Posocco et al., 2021).

A central alternative is the variational characterization of proper calibration errors: $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$1 where the optimal recalibration map is

$$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$2

This turns calibration estimation into a supervised learning problem on prediction space: fit $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$3 to predict $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$4 from $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$5, then estimate the score improvement achieved by recalibration (Berta et al., 27 Feb 2026).

For norm-based errors,

$$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$6

the same paper shows that $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$7 calibration errors can be represented through locally defined proper losses $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$8, even though $$C:=\mathbb{E}[Y\mid f(X)]\in\Delta_k,\qquad C_i=\mathbb{P}(Y=e_i\mid f(X)).$$9 is not induced by a single global proper loss on the simplex. The estimator is implemented by cross-validated fitting of $\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$0; with cross-validation, the estimate is a lower bound in expectation on the true calibration error and avoids the overestimation typical of binning-based procedures (Berta et al., 27 Feb 2026).

A complementary nonparametric route estimates $\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$1 directly with a simplex-adapted kernel conditional expectation estimator, using a Dirichlet kernel. Plugging that estimate into the Bregman formula yields consistent and asymptotically unbiased estimators for all proper calibration errors and refinement terms, including the KL calibration error

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$2

The same framework yields estimators for squared canonical calibration error and for sharpness (Popordanoska et al., 2023).

3. Decision theory, local optimality, and regret

One line of work replaces global risk minimization with a local optimality condition: a predictor is nearly calibrated when its proper loss cannot be improved much by simple post-processing of its outputs. For squared loss, the post-processing gap is

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$3

and the smooth calibration error satisfies

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$4

For general proper losses represented in dual form by a $\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$5-smooth convex function $\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$6, the corresponding inequality is

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$7

These bounds give a quantitative duality between calibration error and local proper-loss optimality under Lipschitz post-processing (Błasiok et al., 2023).

A decision-theoretic formulation defines calibration error as the maximal downstream payoff improvement achievable by recalibrating predictions. In the online binary setting, the paper on calibration for decision making identifies this quantity with Maximum Swap Regret,

$\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$8

where $\mathbb{E}[Y\mid f(X)] = f(X)\quad \text{%%%%2%%%%-almost surely.}$9 is the class of bounded proper scoring rules. This is exactly the supremum, over all payoff-bounded decision tasks, of the payoff gain that could be achieved by recalibrating predictions (Hu et al., 2024).

A related extension defines proper-calibration and proper-calibeating by requiring errors to converge to zero uniformly over all bounded proper scoring rules. In that framework, calibration implies proper-calibration, and proper-calibration is equivalent to universal no regret when best replying to forecasts in decision-making under uncertainty (Foster et al., 26 May 2026). This suggests that proper calibration errors are not only descriptive discrepancies between predictive and empirical distributions; they are also operational regret quantities under all decision problems represented by bounded proper scoring rules.

4. Training losses, recalibration maps, and score-dependent diagnostics

Optimizing a proper loss over all measurable predictors yields the true conditional distribution and therefore perfect calibration, but optimizing a proper loss over a restricted hypothesis class does not in general guarantee calibration. A precise sufficient condition is local optimality with respect to Lipschitz post-processing of predictions or logits: if no simple recalibration map can reduce the proper loss substantially, then smooth calibration error is small (Błasiok et al., 2023).

The relation between training objectives and calibration can be subtle. Cross-entropy is strictly proper and, in the formulation of the focal-loss paper, “implies that the model will yield almost perfect calibration on a training set.” The same paper argues that the generalization gap causes overconfidence on test data and proves that focal loss can be decomposed into a confidence-raising transformation and a proper loss. In the binary case, the focal calibration map is bounded between temperature-scaling maps, and focal temperature scaling combines this map with standard temperature scaling as a post-hoc calibration method (Komisarenko et al., 2024).

Proper calibration errors also make explicit that the best recalibration method depends on the score-induced divergence of interest. Empirical comparisons on CIFAR-10 and CIFAR-100 show that temperature scaling tends to reduce KL-based proper calibration error more than isotonic regression, whereas isotonic regression tends to reduce squared $d:\Delta_k\times\Delta_k\to[0,\infty)$0-type proper calibration error more than temperature scaling (Popordanoska et al., 2023). This suggests that recalibration should be selected relative to the proper score that defines the intended notion of calibration.

From a posterior-evaluation perspective, proper scoring rules evaluate posterior quality, while calibration metrics are diagnostic. In that view, the recommended diagnostic is calibration loss,

$d:\Delta_k\times\Delta_k\to[0,\infty)$1

the reduction in expected proper scoring loss obtained by calibrating the predictions with a chosen transform. This quantity is score-dependent, directly interpretable in units of expected Bayes cost, and is presented as superior to ECE and expected score-divergence calibration metrics for diagnostic use (Ferrer et al., 2024).

5. Noise-limited calibration in gravitational-wave and shear inference

In gravitational-wave detector calibration, “proper calibration errors” are calibration errors that are small enough that they do not dominate the uncertainties in inferred astrophysical parameters. Advanced LIGO reconstructs free-running strain from measured optical power through the response function $d:\Delta_k\times\Delta_k\to[0,\infty)$2: $d:\Delta_k\times\Delta_k\to[0,\infty)$3 Systematic errors in the calibration parameters $d:\Delta_k\times\Delta_k\to[0,\infty)$4 distort both reconstructed strain and noise PSD, and induce parameter biases

$d:\Delta_k\times\Delta_k\to[0,\infty)$5

The criterion for proper calibration is

$d:\Delta_k\times\Delta_k\to[0,\infty)$6

where the statistical uncertainty is limited by the Cramér–Rao bound (Hall et al., 2017).

For a GW150914-like signal in the O1/O2 configuration, the resulting proper calibration requirements are

$d:\Delta_k\times\Delta_k\to[0,\infty)$7

For a higher-power detuned RSE configuration they tighten to

$d:\Delta_k\times\Delta_k\to[0,\infty)$8

The same framework is also applied to tests of a massive graviton, where the calibration requirements remain nearly identical to the massless-graviton case (Hall et al., 2017).

In cosmic shear measurement, the problem is formulated through

$d:\Delta_k\times\Delta_k\to[0,\infty)$9

with multiplicative bias $\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$0, additive bias $\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$1, and noise $\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$2. A first-order “proper” calibration formula avoids division by noisy $\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$3 and instead uses

$\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$4

This calibration removes additive bias on average,

$\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$5

but leaves residual multiplicative bias

$\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$6

The paper concludes that a first-order bias correction is worthwhile in most typical cases, while a higher-order correction is worthwhile only for methods with intrinsically high multiplicative bias ($\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$7 per cent) or when the simulation size is very small ($\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$8 simulated galaxies) (Gillis et al., 2018).

6. Large-scale structure and reciprocity calibration as system-level requirements

For photometric galaxy surveys, calibration error is modeled as a multiplicative sky field

$\mathrm{CE}_d(f)=\mathbb{E}\big[d(f(X),C)\big],$9

where $f(X)=C$0 represents unaccounted-for angular and redshift variations in the selection function. In spherical harmonics, these modes generate both additive contamination and multiplicative coupling between true clustering multipoles, and they generically violate statistical isotropy of the observed galaxy field (Huterer et al., 2012).

The dominant physical sources studied are Galactic dust extinction and spatially varying survey depth. The conversion from magnitude calibration error to number-count calibration error is

$f(X)=C$1

where

$f(X)=C$2

is the faint-end slope of the luminosity function. The paper finds that the largest-angle photometric calibration variations—dipole, quadrupole, and a few more modes, though not the monopole—are the most damaging, and that calibration will need to be understood at the $f(X)=C$3 level, corresponding to rms variations between $f(X)=C$4 and $f(X)=C$5 mag, if biases in dark-energy and primordial-non-Gaussianity parameters are to remain subdominant (Huterer et al., 2012).

In repeater-assisted massive MIMO downlink, calibration errors are reciprocity mismatches among UE RF chains, BS RF chains, and the repeater forward/reverse path: $f(X)=C$6 These factors determine the uplink–downlink relation

$f(X)=C$7

and their residual estimation errors propagate into the effective downlink channel error and the spectral-efficiency denominator through beamforming uncertainty, self-interference, and multi-user interference (Ueda et al., 12 Jun 2026).

The analytical and numerical conclusions are explicit: UE and BS calibration errors degrade spectral efficiency more severely than repeater errors, and proper handling means keeping BS and UE calibration errors small enough that their induced interference terms are negligible at the intended operating point, while moderate repeater calibration errors are more tolerable (Ueda et al., 12 Jun 2026). A plausible implication is that “proper calibration errors” in such systems are best understood not as absolute hardware tolerances, but as operating-region-dependent error budgets defined relative to the system’s target inference or spectral-efficiency criterion.