Averaged Two-Bin Calibration Error

• Averaged Two-Bin Calibration Error is defined as the expected sum of squared biases over two randomized bins, ensuring that reporting true probabilities minimizes the measure.
• It computes unnormalized bias sums over a uniformly random threshold, achieving efficient calibration testing in O(T log T) time for binary probabilistic predictions.
• ATB resolves finite-sample incentive issues by isolating predictor-dependent error from inherent Bernoulli variance, offering a truthful, continuous, and complete calibration metric.

Averaged Two-Bin Calibration Error (ATB) is a calibration measure for binary probabilistic prediction in the batch setting that averages, over a uniformly random threshold $q \in [0,1]$, the squared biases of the two prediction bins $\{v<q\}$ and $\{v\ge q\}$. In the formulation introduced in 2025, ATB was designed to resolve a finite-sample incentive problem in calibration measurement: unlike previously known calibration measures, it is minimized in expectation by reporting the ground-truth probabilities themselves. The same work proves that ATB is truthful, sound, complete, continuous, quadratically related to smooth calibration error and distance to calibration, and computable in $O(T\log T)$ time on a sample of size $T$ (Hartline et al., 18 Aug 2025).

1. Formal definition

The underlying setting admits two equivalent views. In the distribution view, one considers a joint distribution $J$ over $(v,y)\in[0,1]\times\{0,1\}$, where $v$ is a reported probability and $y$ is a binary outcome. In the finite-sequence view, one considers reports $r=(r_1,\dots,r_T)\in[0,1]^T$, ground-truth probabilities $\{v<q\}$0, and realized labels $\{v<q\}$1 with $\{v<q\}$2 independently (Hartline et al., 18 Aug 2025).

For a distribution $\{v<q\}$3 and threshold $\{v<q\}$4, the two bin biases are

$\{v<q\}$5

ATB is then

$\{v<q\}$6

On a finite sequence, the induced empirical form is

$\{v<q\}$7

The same paper also introduces an $\{v<q\}$8 two-bin variant obtained by replacing the squares with absolute values. Jensen’s inequality yields

$\{v<q\}$9

so the squared and $\{v\ge q\}$0 forms are linked up to constant factors (Hartline et al., 18 Aug 2025).

Two aspects are structurally distinctive. First, the partition always has exactly two bins. Second, the threshold is not fixed but averaged uniformly over $\{v\ge q\}$1. ATB is therefore not a single-threshold diagnostic, but an average over all such binary partitions.

2. Truthfulness and finite-sample incentive alignment

The central notion is truthfulness on sequences. A calibration measure $\{v\ge q\}$2 is truthful on sequences if, for every report vector $\{v\ge q\}$3,

$\{v\ge q\}$4

Thus, expected empirical calibration error is minimized by reporting the true probabilities $\{v\ge q\}$5 themselves (Hartline et al., 18 Aug 2025).

ATB satisfies this property exactly. The key identity is the decomposition

$\{v\ge q\}$6

The second term is a predictor-independent variance term. Since $\{v\ge q\}$7 for every $\{v\ge q\}$8 and $\{v\ge q\}$9, one obtains

$O(T\log T)$0

This is the sense in which ATB is perfectly truthful: no approximation, asymptotics, or large-sample caveat is required (Hartline et al., 18 Aug 2025).

The same decomposition yields a stronger rank-preserving property: $O(T\log T)$1 Hence the ordering of predictors by expected empirical ATB coincides exactly with their ordering under the latent quantity $O(T\log T)$2. The paper contrasts this with previously known calibration measures, which incentivize predictors to lie in order to appear more calibrated on a finite sample (Hartline et al., 18 Aug 2025).

This behavior is tied to the absence of per-bin normalization by bin size. ATB aggregates unnormalized squared bin biases, globally normalized by $O(T\log T)$3, and this variance-additive structure is what makes the predictor-dependent term separate cleanly from the Bernoulli noise term.

3. Soundness, completeness, continuity, and zero-calibration characterization

ATB is shown to be sound, complete, and continuous, and these properties are established through constant-factor and quadratic relations to continuous calibration functionals already known in the literature. In particular, for smooth calibration error,

$O(T\log T)$4

and for distance to calibration,

$O(T\log T)$5

Because $O(T\log T)$6 and $O(T\log T)$7 are known to be complete and sound, ATB inherits those properties (Hartline et al., 18 Aug 2025).

These inequalities imply the zero-calibration characterization: $O(T\log T)$8 Thus, although ATB is defined through random two-bin partitions, it does not merely detect a restricted notion of calibration. Zero ATB is equivalent to perfect calibration (Hartline et al., 18 Aug 2025).

The paper also proves Lipschitz continuity. If $O(T\log T)$9 is a coupling of $T$0 with induced marginals $T$1 and $T$2 on $T$3 and $T$4, then

$T$5

In sequence form, for two report vectors $T$6 with common labels,

$T$7

The paper explicitly contrasts this with ECE and binned ECE, which can be discontinuous in the predictions (Hartline et al., 18 Aug 2025).

4. Relation to ECE, smCal, distCal, and earlier two-bin usage

ATB arose against a background in which calibration was often measured by binned expected calibration error (ECE). Standard ECE compares average confidence and empirical accuracy inside bins, and its behavior depends strongly on the binning scheme. Equal-width binning, equal-mass binning, and continuous alternatives all modify this dependence; fixed-bin ECE is sensitive to the number of bins, can ignore empty bins, and exhibits a bias–variance trade-off that scales with the granularity of discretization (Pavlovic, 31 Jan 2025, Futami et al., 2024).

Earlier expository discussions sometimes used “ATB” informally for an ECE-style two-bin construction or for an average over many two-bin threshold splits. In those formulations, ATB was presented either as ECE with $T$8 or as an average of two-bin ECE-type quantities over thresholds (Pavlovic, 31 Jan 2025, Matsubara et al., 2023, Karandikar et al., 2021). The 2025 measure is different. Its bin-level objects are not per-bin accuracy–confidence gaps, but squared unnormalized biases of the form

$T$9

followed by averaging over uniformly random $J$0 (Hartline et al., 18 Aug 2025).

This distinction is substantive rather than terminological. The ECE family was developed as a binned approximation to confidence calibration, and subsequent work emphasized its sensitivity to discretization, instability under rebinning, and lack of consistency unless bins are refined with sample size (Pavlovic, 31 Jan 2025, Futami et al., 2024). ATB, by contrast, was designed to satisfy a truthfulness criterion on finite samples and to admit exact error decompositions unavailable to the standard binned ECE construction (Hartline et al., 18 Aug 2025).

Relative to continuous measures, ATB sits between smooth calibration error and distance to calibration. The paper proves that the $J$1 two-bin error is a constant-factor approximation to both $J$2 and $J$3, and that ATB is quadratically related to them. This is the basis for both its theoretical guarantees and its algorithmic utility (Hartline et al., 18 Aug 2025).

5. Computation and calibration testing

ATB is efficient to compute on a sample of size $J$4. The algorithm sorts the reports $J$5, computes prefix and suffix sums of $J$6, and exploits the fact that the two bin biases are constant on intervals between consecutive sorted report values. If $J$7 and $J$8, then

$J$9

so after sorting, the integration over thresholds is a linear scan. The total running time is $(v,y)\in[0,1]\times\{0,1\}$0 (Hartline et al., 18 Aug 2025).

This complexity compares favorably with existing continuous measures. The paper states that computing $(v,y)\in[0,1]\times\{0,1\}$1 currently costs $(v,y)\in[0,1]\times\{0,1\}$2, while computing $(v,y)\in[0,1]\times\{0,1\}$3 costs $(v,y)\in[0,1]\times\{0,1\}$4. ATB therefore provides a simpler and faster route to quantities that are provably linked to both (Hartline et al., 18 Aug 2025).

The same paper analyzes calibration testing through validity with respect to distance to calibration. ATB is $(v,y)\in[0,1]\times\{0,1\}$5-valid with $(v,y)\in[0,1]\times\{0,1\}$6, and this rate is stated to be information-theoretically optimal. Concretely, for calibrated $(v,y)\in[0,1]\times\{0,1\}$7, with high probability,

$(v,y)\in[0,1]\times\{0,1\}$8

while for miscalibrated $(v,y)\in[0,1]\times\{0,1\}$9 with $v$0, with high probability,

$v$1

for a sufficiently large constant $v$2 (Hartline et al., 18 Aug 2025).

The computational consequence is that ATB yields faster calibration tests for $v$3 and distance to calibration than direct computation of those measures. The paper explicitly frames this as an improvement in running time and simplicity for the calibration testing problem studied by Hu et al. (2024) (Hartline et al., 18 Aug 2025).

6. UBSEs and the broader construction principle

ATB is the simplest instance of a broader family introduced in the same work: Unnormalized Binned Squared Errors (UBSEs). In a UBSE, the indices $v$4 are partitioned into bins $v$5; the partition may be randomized and may depend on the reports $v$6, but not on the ground-truth probabilities $v$7. For each bin,

$v$8

and the calibration functional is

$v$9

ATB corresponds to the special case $y$0, with

$y$1

and $y$2 (Hartline et al., 18 Aug 2025).

The crucial structural constraint is that UBSEs use no per-bin normalization by bin size. The paper identifies this as the mechanism behind truthfulness: variance additivity for independent Bernoulli outcomes produces the decomposition

$y$3

with a predictor-independent second term. ATB is the most elementary nontrivial member of this class (Hartline et al., 18 Aug 2025).

The UBSE framework also yields other truthful measures. The paper gives quantile-binned $y$4-ECE as an example: if one partitions the sorted reports into equal-size bins and aggregates unnormalized squared bin biases rather than normalized per-bin gaps, the resulting measure is truthful. This suggests that ATB is not an isolated construction but the canonical minimal instance of a larger recipe for incentive-compatible calibration measurement (Hartline et al., 18 Aug 2025).

Historically, this marks a shift from viewing two-bin procedures as merely coarse discretizations of ECE toward viewing them as components of a truthful batch calibration theory. Earlier work on calibration diagnostics emphasized the limitations of coarse binning, bin-choice sensitivity, and the bias inherent in histogram estimators (Pavlovic, 31 Jan 2025, Futami et al., 2024). ATB retains the simplicity of two-bin partitions, but recasts them in a form that supports exact finite-sample incentive alignment, continuity, and efficient testing (Hartline et al., 18 Aug 2025).