Calibeating: Adaptive Forecast Post-Processing

• Calibeating is a post-processing paradigm in online probabilistic forecasting that decouples calibration from sharpness to enhance predictive informativeness.
• It adjusts a stream of potentially uncalibrated forecasts using historical outcomes through bin-wise, stochastic, or continuous methods to minimize cumulative Brier score.
• The framework leverages online regret minimization techniques to offer strong theoretical guarantees, achieving sublinear or logarithmic error rates even in multi-expert settings.

Calibeating is a post-processing paradigm in online probabilistic forecasting that enables an adaptive learner to “beat” an external forecaster's Brier score without sacrificing refinement, and, crucially, without degrading the underlying predictive informativeness. The calibeating approach operates under a very general online learning framework: it takes as input a stream of probabilistic forecasts from a potentially uncalibrated source (or multiple sources), and—using information about observed outcomes—outputs its own sequence of forecasts whose cumulative loss is bounded by the refinement component of the original forecaster(s) plus a sublinear or logarithmic remainder. The paradigm provides a principled mechanism to decouple calibration and sharpness (expertise), yielding a theory and algorithms for post-hoc calibration that preserve or improve informativeness while rigorously controlling miscalibration.

1. Formal Framework for Calibeating

Calibeating is defined in terms of the decomposition of proper loss (most notably, the Brier score) into calibration and refinement. For a stream of outcomes $a_s \in A$ and forecasts $c_s \in C \subset \mathbb{R}^m$, the following key quantities are considered:

• Brier score: $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$
• Calibration score: $$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$, where $$\bar{a}_t(x) = \frac{1}{n_t(x)} \sum_{s: c_s = x} a_s$$
• Refinement score: $$R_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - \bar{a}_t(c_s)\|^2$$

These decompose the cumulative loss as $B_t = K_t + R_t$. Calibeating aims to post-process any forecast sequence $b$ into a new sequence $c$ such that

$B_t(c, a) \leq R_t(b, a) + \varepsilon_t$

where $c_s \in C \subset \mathbb{R}^m$0 as $c_s \in C \subset \mathbb{R}^m$1 (Foster et al., 2022, Chen et al., 23 Mar 2026).

2. Algorithms for Calibeating

Bin-wise Calibeating

The core procedure is bin-wise post-processing: for each forecast value $c_s \in C \subset \mathbb{R}^m$2, the calibeating forecaster $c_s \in C \subset \mathbb{R}^m$3 emits the empirical frequency of success among prior events with the same forecast bin. Formally,

$c_s \in C \subset \mathbb{R}^m$4

This approach provably guarantees $c_s \in C \subset \mathbb{R}^m$5 for a finite forecast set $c_s \in C \subset \mathbb{R}^m$6 (Foster et al., 2022).

Stochastic and Continuous Extensions

• Stochastic calibeater: For continuous or large $c_s \in C \subset \mathbb{R}^m$7, a stochastic procedure samples from a small grid in each bin and solves an explicit minimax problem to ensure both $c_s \in C \subset \mathbb{R}^m$8 and calibration in expectation (Foster et al., 2022).
• Continuous deterministic variant: A deterministic, continuously calibrated calibeater is constructed via fixed-point arguments guaranteeing instantaneous gain $c_s \in C \subset \mathbb{R}^m$9 across bins, thus offering fine-grained calibration and calibeating simultaneously (Foster et al., 2022).

3. General Theory: Equivalence to Regret Minimization

The reduction in "Calibeating Made Simple" shows calibeating for any proper loss is minimax-equivalent to online regret minimization, with the calibeating regret determined up to constants by the complexity of the underlying loss (Chen et al., 23 Mar 2026):

• For a mixable loss (Brier, log), calibeating rate is $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$0, where $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$1 is the set of unique external forecasts over $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$2 rounds.
• For bounded proper losses, the rate is $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$3.
• The equivalence is proved by aggregating no-regret subroutines per distinct forecast bin and establishing matching lower bounds via block construction (Chen et al., 23 Mar 2026).

The theory also applies to multi-calibeating, requiring the algorithm to simultaneously calibeat $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$4 external forecasters. The optimal multi-calibeating rate is the sum of the best calibeating rate and the classical expert regret rate, e.g., $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$5 in the mixable case (Chen et al., 23 Mar 2026).

4. Simultaneous Calibeating and Calibration

Prior constructions did not always yield algorithms which are both calibrated (small $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$6) and calibeating (small difference from the reference refinement). Recent results show that for Brier loss with binary outcomes, one can simultaneously achieve the optimal calibeating rate and sublinear calibration error (Chen et al., 23 Mar 2026):

• By discretizing predictions onto a grid and employing swap-regret minimization (Blackwell approachability) along with lopsided expert aggregation, a metaalgorithm is constructed.
• For $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$7, setting grid size $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$8 yields $$B_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|a_s - c_s\|^2$$9 calibeating and $$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$0 calibration error (Chen et al., 23 Mar 2026).

5. Concrete Instantiations and Practical Algorithms

Several practical algorithms have been proposed and empirically validated:

• Tracking and Hedging: The TOPS (tracking) variant directly bins the base forecaster’s output and emits empirical frequencies; the HOPS (hedging) variant incorporates randomized calibration within bins for adversarial robustness (Gupta et al., 2023).
• Online Platt Scaling with Calibeating: When standard online calibration is miscalibrated due to distribution drift or nonstationarity, stacking calibeating on top restores calibration and minimizes Brier (or log) regret without degrading sharpness (Gupta et al., 2023).
• Extension to Beta Scaling: The framework generalizes to non-sigmoid recalibrators, supporting online adaptation and calibeating over richer probability transformations (Gupta et al., 2023).

6. Evaluation and Theoretical Guarantees

Calibeating algorithms guarantee, under broad conditions, that cumulative loss is bounded above by the best achievable refinement score of the reference forecaster plus algorithmically optimal rates (logarithmic or square-root, depending on loss class) (Chen et al., 23 Mar 2026). Empirically, calibeating consistently reduces calibration error and Brier score over static or naive recalibration methods, especially in non-i.i.d. or distributionally drifting data. In adversarial settings, the randomized (hedging) versions retain theoretical calibration guarantees.

The following table summarizes core theoretical rates for calibeating with various loss classes as established in (Chen et al., 23 Mar 2026):

Loss class	Calibeating rate $$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$1	Multi-calibeating rate (N experts)
Mixable (Brier, log)	$$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$2	$$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$3
Bounded proper	$$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$4	$$K_t(c, a) = \frac{1}{t} \sum_{s=1}^t \|\bar{a}_t(c_s) - c_s\|^2$$5

7. Testing Calibeating: Detection and Statistical Significance

Techniques such as T-Cal provide minimax-optimal hypothesis testing for model calibration, directly probing whether reductions in calibration error reflect genuine improvements or are artifacts due to overfitting or random fluctuations (Lee et al., 2022). T-Cal avoids the pitfall where manipulations that simply shrink the empirical ECE (as some “calibeating” post-processors might) might not improve the true calibration curve. Adaptive tests assess whether calibeating introduces false improvements and rigorously quantify the detectability of miscalibration as a function of sample size and smoothness.

• "Calibeating: Beating Forecasters at Their Own Game" (Foster et al., 2022)
• "Online Platt Scaling with Calibeating" (Gupta et al., 2023)
• "Calibeating Made Simple" (Chen et al., 23 Mar 2026)
• "T-Cal: An optimal test for the calibration of predictive models" (Lee et al., 2022)