Adaptive Calibration in Non-Stationary Environments

Abstract: Making calibrated online predictions is a central challenge in modern AI systems. Much of the existing literature focuses on fully adversarial environments where outcomes may be arbitrary, leading to conservative algorithms that can perform suboptimally in more benign settings, such as when outcomes are nearly stationary. This gap raises a natural question: can we design online prediction algorithms whose calibration error automatically adapts to the degree of non-stationarity in the environment, smoothly interpolating between i.i.d. and adversarial regimes? We answer this question in the affirmative and develop a suite of algorithms that achieve adaptive calibration guarantees under multiple calibration measures. Specifically, with $T$ being the number of rounds and $C\in[0,T]$ being an unknown non-stationary measure defined as the minimal $\ell_1$ deviation of the mean outcomes, our algorithms attain $\widetilde{O}(\sqrt{T}+(TC)^{\frac{1}{3}})$ for $\ell_1$ calibration error and $\widetilde{O}((1+C)^{\frac{1}{3}})$ for both $\ell_2$ and pseudo KL calibration error. These bounds match the optimal rates in the stationary case ($C=0$) and recover known guarantees in the fully adversarial regime ($C=T$). Our approach builds on and extends prior work [Hu et al., 2026, Luo et al., 2025], introducing an epoch-based scheduling together with a novel non-uniform partition of the prediction space that allocates finer resolution near the underlying ground truth.

• The paper presents an adaptive calibration algorithm that interpolates error bounds between stationary and adversarial regimes using non-uniform partitioning.
• It employs an epoch-based framework with doubling intervals and local probability estimation to dynamically adjust prediction bins to shifting data distributions.
• Numerical results validate that the adaptive approach improves calibration in online prediction, with significant implications for areas such as medical diagnosis and risk forecasting.

Adaptive Calibration Algorithms for Non-Stationary Online Prediction

Problem Formulation and Motivation

The paper "Adaptive Calibration in Non-Stationary Environments" (2605.11490) addresses the challenge of maintaining calibrated probabilistic predictions in sequential decision-making scenarios where the outcome distribution may evolve over time. Calibration requires predicted probabilities to match empirical frequencies; for instance, events assigned probability $p$ must occur with frequency $p$. Previous work has focused primarily on adversarial settings where outcomes are arbitrary, resulting in highly robust but often conservative algorithms. These algorithms fail to exploit structure in more benign, stationary, or slowly changing environments, leading to suboptimal calibration error rates when data is less adversarial.

A central question is whether online prediction algorithms can adapt their calibration guarantees to the level of non-stationarity in the environment, i.e., interpolate between the i.i.d. regime (stationary outcomes) and fully adversarial regime. The paper answers this question affirmatively and proposes algorithms that achieve calibration error bounds depending explicitly on an unknown non-stationarity measure $C$, defined as the minimum $\ell_1$ deviation of outcome means from their median, with $C=0$ representing the stationary regime and $C=T$ the fully adversarial regime.

Calibration Measures and Non-Stationarity Metric

The paper focuses on three calibration metrics:

• $\ell_1$ Calibration Error ($Cal_1$): Measures absolute deviation between the predicted probability and empirical frequency across prediction bins.
• $\ell_2$ Calibration Error ($Cal_2$): Measures squared deviation, easier to optimize and still enables strong downstream properties such as swap regret.
• Pseudo KL Calibration Error ($p$0): Uses the KL divergence, strictly stronger than $p$1 and relevant for broader proper loss applications.

For each, the paper develops matching lower and upper bounds in the stationary case ($p$2), establishing baselines for adaptation. In particular, optimal rates are $p$3 for $p$4 and $p$5 for $p$6 and $p$7.

Crucially, the non-stationarity metric $p$8 is the total minimal $p$9 deviation of mean outcomes from their global median:

$C$0

This quantity inherently appears in concentration and deviation bounds, making it a natural control for adaptive algorithms.

Algorithmic Framework and Analytical Innovations

The core contributions are an adaptive framework employing non-uniform partitioning of the prediction space (see illustration below):

Figure 1: Illustration of the non-uniform partition created by the adaptive epoch-based framework, allocating finer and denser prediction bins near the ground truth probability.

The algorithm operates in epochs with doubling length. In each epoch:

• It estimates the local ground truth probability from the previous epoch.
• It constructs a non-uniform partition, assigning finer resolution near this estimated probability and coarser partitions elsewhere.
• It runs a calibration algorithm within these partitions, allowing specialization to the local stationarity.
• This epoch schedule tracks non-stationarity by updating the partitioning as the empirical frequency shifts over time.

The authors show, through novel analytic arguments, that adaptive calibration error bounds are attainable:

• For $C$1, the modified algorithm achieves $C$2, interpolating between the best stationary and adversarial rates.
• For $C$3 and $C$4, the epoch-based framework with non-uniform partitioning produces $C$5, matching optimal stationary ($C$6) and adversarial ($C$7) bounds.

Numerical results demonstrate that these rates recover existing minimax optimality in extremes and provide smooth interpolation in intermediate regimes, with computational complexity per round scaling as $C$8.

Analytical Results and Strong Claims

The analytical results are supported by careful lower bound reductions for stationary cases, as well as refined upper bounds based on visit frequency and error decomposition across non-uniform partitions.

Key claims include:

• The adaptive bounds $C$9 for $\ell_1$0 and $\ell_1$1 for $\ell_1$2 and $\ell_1$3 are not attainable by uniform partitioning or globally parameterized adversarial algorithms; the non-uniform approach is critical.
• The framework is versatile: it can recover $\ell_1$4 bounds with improved computational efficiency in stationary regimes.
• The doubling trick adaptation allows the algorithms to function even without prior knowledge of $\ell_1$5, incurring only a logarithmic overhead in calibration error.
• The epoch-based adaptivity and non-uniform partitioning may extend to other notions of calibration, such as smooth calibration and calibration decision loss.

Practical and Theoretical Implications

Practically, these results enable more efficient, less conservative calibration in online AI systems operating in dynamic environments, with immediate impact in applications like medical diagnosis and risk forecasting where calibration error is critical. Theoretically, the paper advances the understanding of optimal calibration in non-stationary regimes, bridging the gap between adversarial and i.i.d. settings, and providing tools for adaptive algorithms in sequential prediction.

The framework's modularity suggests potential applicability to other calibration metrics and more structured settings (contextual, multiclass, etc.), offering a promising direction for future research on adaptive online learning.

Conclusion

The paper establishes that adaptive calibration in online prediction is achievable, with error bounds smoothly interpolating between stationary and adversarial regimes based on an intrinsic measure of non-stationarity. The epoch-based framework with non-uniform partitioning represents a significant advance in both algorithm design and analytic understanding, providing a unifying approach for calibration across diverse environments. Future work involves tightening intermediate lower bounds, extending the framework to more general prediction settings, and exploring its applicability to other calibration notions.