Truthful Calibration Measures

• Truthful calibration measures are defined to ensure that reporting true probabilities minimizes empirical miscalibration, even in finite sample scenarios.
• They incorporate mechanism design principles by distinguishing properties such as soundness, completeness, continuity, and testability beyond classical ECE.
• Methods like ATB, classwise ℓ₂-ECE, and SSCE address shortcomings of traditional metrics by aligning calibration with decision-theoretic and sequential forecasting needs.

Truthful calibration measures are calibration errors designed so that, in expectation, reporting the ground-truth probabilities minimizes empirical miscalibration. This criterion strengthens the classical population definition of calibration—conditional unbiasedness of predictive probabilities—by adding an incentive-compatibility requirement at finite sample size. The recent literature shows that these requirements are distinct: a measure may be sound, complete, or consistent as a distance to calibration, yet still be non-truthful because a forecaster can obtain a lower expected score by strategically distorting probabilities. The resulting theory treats calibration not only as a statistical property but also as a mechanism-design problem, and increasingly describes it through an indistinguishability lens comparing the world hypothesized by the predictor with the realized world (Haghtalab et al., 2024, Hartline et al., 18 Aug 2025, Gopalan et al., 2 Sep 2025).

1. Formal definitions and desiderata

In the binary setting, perfect calibration means that for a predictor $F:\mathcal X \to [0,1]$,

$\mathbb{E}[Y \mid F(X)=v] = v$

for every forecast value $v$ in the support of $F(X)$. In the multiclass setting, calibration can be stated coordinatewise; under inherent human disagreement, the ideal becomes instance-level matching of the human label distribution, $f_c(x)=\Pr(Y=c\mid X=x)$ (Błasiok et al., 2022, Baan et al., 2022).

The finite-sample truthfulness criterion is stronger. In the batch multiclass formulation, a calibration measure

$$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$

is truthful if, for any ground-truth distributions $p_1^*,\dots,p_n^*\in \Delta(Y)$ and any alternative reports $p_1,\dots,p_n$, one has

$$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$

where $y_i\sim p_i^*$ i.i.d. Equivalently, the predictor minimizes expected calibration error by “telling the truth” (Lu et al., 7 Oct 2025).

Sequential work uses an approximate notion. If $\mathbb{E}[Y \mid F(X)=v] = v$0 is the honest forecaster that predicts the true conditional probability at each round, and $\mathbb{E}[Y \mid F(X)=v] = v$1 is the best achievable expected penalty under calibration measure $\mathbb{E}[Y \mid F(X)=v] = v$2, then $\mathbb{E}[Y \mid F(X)=v] = v$3 is $\mathbb{E}[Y \mid F(X)=v] = v$4-truthful if

$\mathbb{E}[Y \mid F(X)=v] = v$5

This definition allows truthful forecasting to be optimal up to constant or polylogarithmic factors rather than exactly (Haghtalab et al., 2024).

Truthfulness is only one desideratum. The contemporary literature repeatedly distinguishes it from soundness, completeness, continuity, consistency, testability, and actionability. A particularly influential benchmark is the $\mathbb{E}[Y \mid F(X)=v] = v$6 distance to the nearest perfectly calibrated predictor,

$\mathbb{E}[Y \mid F(X)=v] = v$7

which is presented as a ground-truth notion of distance from calibration in the binary prediction-only access model (Błasiok et al., 2022).

2. Why classical calibration measures are insufficient

The standard expected calibration error (ECE) bins forecast values and compares average confidence with average accuracy. In its common top-label form,

$\mathbb{E}[Y \mid F(X)=v] = v$8

For multiclass models, variants differ in whether they use only the maximum probability or all class probabilities, whether they threshold low scores, whether they condition on class, whether bins are uniform or adaptive, and whether the discrepancy is measured with an $\mathbb{E}[Y \mid F(X)=v] = v$9 or $v$0 norm (Nixon et al., 2019).

These design choices are not innocuous. Across MNIST, Fashion-MNIST, CIFAR-10/100, and ImageNet, the rank ordering of recalibration methods changes drastically with the calibration measure. The same study recommends conditioning on the class, using adaptive equal-count bins, and using the $v$1 norm rather than the $v$2 norm, because these choices yield more effective calibration evaluations and more stable method rankings (Nixon et al., 2019).

The truthfulness literature isolates a sharper failure mode: classical measures can reward dishonest reporting. In sequential prediction, ECE, smooth calibration error, distance from calibration, interval calibration, kernel calibration, and $v$3-calibration all admit large truthfulness gaps under a triple-block construction, whereas proper scoring rules are perfectly truthful but not complete because they incur $v$4 penalty on an honest i.i.d. sequence (Haghtalab et al., 2024). In the batch setting, standard binned ECE is explicitly contrasted with truthful measures by noting that a dishonest constant predictor can achieve lower expected error than the Bayes-optimal report (Lu et al., 7 Oct 2025).

Further deficiencies appear when testability and decision-making are taken seriously. ECE is actionable for binary threshold decisions, but it is not testable in a distribution-free sense for general smooth or strictly increasing predictors; by contrast, distance from calibration is testable at the $v$5 rate but is not actionable, because arbitrarily small $v$6 can coexist with catastrophically suboptimal threshold decisions (Rossellini et al., 27 Feb 2025). The binary distance-from-calibration theory also shows that popular measures such as ECE fail basic properties like continuity, while smooth, interval, and Laplace-kernel calibration do satisfy polynomial relations to $v$7 (Błasiok et al., 2022).

3. Perfectly truthful measures in the batch setting

The first explicitly perfectly truthful batch construction in the binary setting is Averaged Two-Bin calibration error (ATB). For a report $v$8, outcomes $v$9, and threshold $F(X)$0, define

$F(X)$1

ATB is

$F(X)$2

ATB is a special case of an Unnormalized Binned Squared Error (UBSE): partition indices into bins depending on the report, compute raw bias in each bin, square, and average. The key decomposition is

$F(X)$3

which implies truthfulness because $F(X)$4 and vanishes at $F(X)$5. ATB is further stated to be sound, complete, continuous, and computable in $F(X)$6, and it satisfies quadratic relations to smooth calibration and distance to calibration: $F(X)$7 (Hartline et al., 18 Aug 2025).

The same UBSE recipe yields other perfectly truthful measures. One example is quantile-binned $F(X)$8-ECE: sort reports, partition them into equal-sized consecutive bins, and sum squared raw residuals over bins. The construction is perfectly truthful because the same variance-additivity argument applies; if the number of bins grows while the bin fraction vanishes, the resulting error is also consistent (Hartline et al., 18 Aug 2025).

The multiclass extension proceeds through class-wise aggregation. For each class $F(X)$9, form the binary subproblem $f_c(x)=\Pr(Y=c\mid X=x)$0, apply a truthful binary measure coordinatewise, and average over classes. Using quantile-binned squared ECE gives a classwise $f_c(x)=\Pr(Y=c\mid X=x)$1-ECE that is truthful for every sample size $f_c(x)=\Pr(Y=c\mid X=x)$2 and every bin count $f_c(x)=\Pr(Y=c\mid X=x)$3. The same paper proves a dominance-preserving robustness theorem: if two calibrated predictors are ordered by every proper single-sample loss, then the expected ordering induced by classwise $f_c(x)=\Pr(Y=c\mid X=x)$4-ECE never flips as $f_c(x)=\Pr(Y=c\mid X=x)$5 varies. Confidence aggregation, by contrast, does not in general preserve truthfulness, because a predictor can lie about which class is most likely in order to reduce measured error (Lu et al., 7 Oct 2025).

A practical implication is that the usual confidence-only reduction from multiclass to binary calibration is not merely incomplete; it can invalidate the truthfulness guarantee itself. In the truthfulness framework, classwise aggregation is the safe extension and confidence aggregation is the risky one (Lu et al., 7 Oct 2025).

4. Sequential truthfulness and decision-theoretic calibration

The sequential literature begins from a negative result: existing calibration measures are far from truthful. The main positive construction in this line is the Subsampled Smooth Calibration Error (SSCE),

$f_c(x)=\Pr(Y=c\mid X=x)$6

The intuition stated in the original work is that subsampling destroys the ability to perfectly game the statistic on small bias patterns. SSCE is complete and sound, and it is $f_c(x)=\Pr(Y=c\mid X=x)$7-truthful for a universal constant $f_c(x)=\Pr(Y=c\mid X=x)$8. The proof uses martingale concentration, a realized-variance process, and a lower bound showing that any forecaster must accumulate enough large absolute errors to pay comparable SSCE (Haghtalab et al., 2024).

A related construction targets decision-theoretic guarantees. Step calibration error is

$f_c(x)=\Pr(Y=c\mid X=x)$9

and the subsampled version averages this quantity over uniformly random subsets of rounds. The subsampled step calibration measure is shown to be decision-theoretic because it upper-bounds $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$0, the external-regret-based calibration quantity connected to best-responding downstream agents. It is also truthful up to an $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$1 factor on product distributions and up to an $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$2 factor in a $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$3-smoothed setting (Qiao et al., 4 Mar 2025).

The same work proves an impossibility result for the non-smoothed case: any calibration measure that is both complete and decision-theoretic must be discontinuous and non-truthful. This establishes a structural tension. A plausible implication is that, in sequential settings, truthfulness and decision-theoretic guarantees may have to be traded off unless one assumes product structure, smoothing, or a weakened notion of actionability (Qiao et al., 4 Mar 2025).

5. Testability, actionability, and robustness

A central binary decision-theoretic development is Cutoff Calibration Error, defined by

$$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$4

The motivation is that many decisions depend only on whether a forecast falls below or above a threshold. The measure is testable with a plug-in estimator that scans over intervals, and it is actionable in the sense that controlling CCE bounds excess risk against all monotonic post-processors of the score. The stated guarantee is

$$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$5

while the empirical estimator enjoys a high-probability $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$6 deviation bound (Rossellini et al., 27 Feb 2025).

The same paper places ECE and $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$7 on opposite sides of a testability-actionability divide: ECE is actionable but not testable, whereas $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$8 is testable but not actionable. Cutoff Calibration Error is introduced specifically to bridge this gap for threshold-type decisions (Rossellini et al., 27 Feb 2025).

A later construction addresses the stronger requirement of full swap regret. Soft-Binned Calibration Decision Loss (SCDL) uses randomized rounding to a grid, evaluates a soft-binned decision loss at that resolution, and then takes the best dyadic grid up to the tradeoff term $$\mathrm{Cal}:(\Delta(Y))^n \times Y^n \to \mathbb{R}$$9. SCDL is stated to be fully actionable without weakening the full-swap requirement, testable with nearly optimal error rate, continuous, and consistent. Its testability guarantee is

$p_1^*,\dots,p_n^*\in \Delta(Y)$0

with high probability, and any calibration measure with the same actionability guarantee must incur $p_1^*,\dots,p_n^*\in \Delta(Y)$1 estimation error (Bairaktari et al., 18 May 2026).

The current landscape can therefore be summarized as follows.

Measure	Setting	Properties stated in the literature
ATB	Batch binary	Perfectly truthful, sound, complete, continuous (Hartline et al., 18 Aug 2025)
Classwise $p_1^*,\dots,p_n^*\in \Delta(Y)$2-ECE	Batch multiclass	Perfectly truthful; dominance-preserving robustness (Lu et al., 7 Oct 2025)
SSCE	Sequential	Complete, sound, $p_1^*,\dots,p_n^*\in \Delta(Y)$3-truthful (Haghtalab et al., 2024)
StepCE$p_1^*,\dots,p_n^*\in \Delta(Y)$4	Sequential	Decision-theoretic; truthful up to $p_1^*,\dots,p_n^*\in \Delta(Y)$5 or $p_1^*,\dots,p_n^*\in \Delta(Y)$6 factors (Qiao et al., 4 Mar 2025)
Cutoff Calibration Error	Binary threshold decisions	Testable and actionable (Rossellini et al., 27 Feb 2025)
SCDL	Full swap regret	Fully actionable, testable with nearly optimal error rate, continuous, consistent (Bairaktari et al., 18 May 2026)

6. Human disagreement, general output spaces, and broader formulations

Truthfulness also depends on what counts as the target distribution. When human annotators inherently disagree, measuring calibration against a single majority label is theoretically problematic. For an instance $p_1^*,\dots,p_n^*\in \Delta(Y)$7 with empirical human distribution $p_1^*,\dots,p_n^*\in \Delta(Y)$8 and model output $p_1^*,\dots,p_n^*\in \Delta(Y)$9, the proposed instance-level measures are: class-frequency calibration

$p_1,\dots,p_n$0

ranking calibration

$p_1,\dots,p_n$1

and entropy calibration

$p_1,\dots,p_n$2

These measures directly compare model probabilities to the full human vote distribution rather than to a majority-vote gold label. On ChaosNLI, an oracle $p_1,\dots,p_n$3 has ECE $p_1,\dots,p_n$4 but mean DistCE $p_1,\dots,p_n$5, RankCS $p_1,\dots,p_n$6, and mean absolute entropy error $p_1,\dots,p_n$7, illustrating the pathology of classical ECE under disagreement (Baan et al., 2022).

For probabilistic models beyond classification, kernel-based calibration provides a different generalization. The general framework compares the joint laws of $p_1,\dots,p_n$8 and $p_1,\dots,p_n$9, where $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$0, through an integral probability metric over test functions on $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$1. With a characteristic kernel, the resulting Kernel Calibration Error is zero if and only if the model is calibrated. The framework supplies unbiased and low-variance estimators, together with asymptotic tests for calibration, and applies to arbitrary predictive output spaces including real-valued and vector-valued regression targets (Widmann et al., 2022).

In multiclass classification, matrix-valued kernels play a related role. The multi-class Kernel Calibration Error generalizes ECE, MCE, and MMCE, admits unbiased and consistent estimators through U-statistics, and yields calibration tests with p-value bounds and asymptotic null approximations. This gives calibration measures a meaningful unit and scale by tying them directly to hypothesis testing rather than to raw histogram discrepancies (Widmann et al., 2019).

The broadest synthesis is the indistinguishability viewpoint. Let $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$2 be the joint law of the predictor’s score and the real label, and let $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$3 be the corresponding joint law in the predictor’s hypothetical world. A calibration measure is then the distinguishing advantage of a class of tests between $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$4 and $$\mathbb{E}\bigl[\mathrm{Cal}(p_1^*,\dots,p_n^*;y_1,\dots,y_n)\bigr] \le \mathbb{E}\bigl[\mathrm{Cal}(p_1,\dots,p_n;y_1,\dots,y_n)\bigr],$$5. Choosing all bounded distinguishers yields ECE; choosing Lipschitz distinguishers yields smooth calibration error; choosing RKHS test functions yields kernel calibration; choosing decision-task distinguishers yields decision-theoretic calibration loss. This perspective clarifies why different measures emphasize different desiderata—truthfulness, continuity, sample complexity, or downstream utility—and why no single measure simultaneously trivializes all tradeoffs (Gopalan et al., 2 Sep 2025).