Multi-Forecaster Calibeating Framework

• The paper introduces a framework that aggregates multiple forecasters to achieve a cumulative loss lower than that of any single predictor by aligning calibration with refinement.
• The methodology employs deterministic and stochastic binning along with online regret minimization to guarantee minimax optimality and improved Brier score performance.
• Practical implementations utilize ensemble techniques such as bagging and boosting, demonstrating significant enhancements in forecast accuracy and robustness in adversarial settings.

Multi-forecaster calibeating is a framework for aggregating multiple probability forecasters to produce an ensemble whose cumulative loss is strictly less than that of any constituent forecaster, as measured relative to an informativeness-based benchmark (refinement). This objective combines calibration—statistical agreement between predicted and observed probabilities—with the notion of beating each external forecaster’s Brier score by leveraging refinement. The central challenge is to achieve this in an online, possibly adversarial, setting, with guarantees that are robust to the behavior of the input forecasters.

1. Calibration, Refinement, and the Calibeating Principle

Classic forecast evaluation is based on proper scoring rules, such as the Brier score, which decomposes into calibration and refinement:

• Calibration ($K_t$) measures the squared error between forecasted probabilities and empirical frequencies.
• Refinement ($R_t$) quantifies the variance (informativeness) of the forecast, reflecting the ability to sort outcomes into bins with consistent outcomes.
• With Brier score $B_t$, the decomposition is $B_t = K_t + R_t$ (Foster et al., 2022).

A forecast is calibrated if $K_t \to 0$ as $t \to \infty$, but this is insufficient for measuring expertise: perfect calibration can be trivially achieved by always forecasting the base rate, yielding zero refinement. Calibeating addresses the inverse problem: constructing a procedure that is at least as calibrated as the original forecasters while achieving lower Brier loss (better informativeness).

2. Multi-Forecaster Calibeating: Definitions and Guarantees

Given $K$ external forecasters issuing predictions $b_t^{(n)}$ from finite sets $B_n$ ($n=1,\dots,K$), multi-forecaster calibeating constructs an aggregate $R_t$0 that outperforms all. The guarantee is: $R_t$1 where $R_t$2 denotes the joint index set of all bins, and $R_t$3 and $R_t$4 are the ensemble’s Brier and each forecaster’s refinement scores, respectively (Foster et al., 2022).

The algorithmic solution is an extension of binning-based procedures: predictions are generated by maintaining empirical averages per joint bin $R_t$5 and updating based on observed outcomes. The procedure can be made deterministic or stochastic, with stochastic variants capable of achieving calibration as well as the calibeating property (Foster et al., 2022).

3. Minimax Optimality and Regret-Reduction View

Recent work reduces multi-forecaster calibeating to classical regret minimization in online learning, demonstrating that it is minimax-equivalent to a two-part problem:

• Per-forecaster calibeating subproblems (beat each external forecaster up to their refinement score).
• Expert aggregation (compete with the best among $R_t$6 forecasters) (Chen et al., 23 Mar 2026).

Let $R_t$7 be the number of forecasters, $R_t$8 the maximum number of distinct forecast values, and $R_t$9 the time horizon. The minimax excess loss is

$B_t$0

where $B_t$1 and $B_t$2 are the minimax risks for single-forecaster calibeating and expert aggregation, respectively (Chen et al., 23 Mar 2026).

For mixable losses, including Brier and log loss, this yields rate-optimal performance:

• Single-forecaster: $B_t$3.
• Expert aggregation: $B_t$4.
• Multi-calibeating: $B_t$5 excess over the refinement of every forecaster (Chen et al., 23 Mar 2026).

The algorithm combines $B_t$6 copies of a calibeating subroutine (one per forecaster) with a standard expert algorithm (e.g., Hedge). At each round, candidate predictions from each calibeating subroutine are presented to the expert algorithm, which determines the final aggregate output (Chen et al., 23 Mar 2026).

4. Methodologies: Deterministic, Stochastic, and Continuously Calibrated Aggregation

Deterministic Binning (DetCalibeat):

Bins over $B_t$7 are initialized with counts and cumulative sums. For each incoming multi-forecaster bin $B_t$8, the ensemble forecast is the empirical mean of previous outcomes for $B_t$9 if observed; otherwise, any point in $B_t = K_t + R_t$0 is chosen. This yields the stated $B_t = K_t + R_t$1 upper bound (Foster et al., 2022).

Stochastic Calibrated Aggregation:

By discretizing the forecast space to a finite $B_t = K_t + R_t$2-grid and employing a minimax outgoing theorem, one constructs a stochastic procedure sampling aggregate forecasts with the property: $B_t = K_t + R_t$3 This method guarantees both calibration (small $B_t = K_t + R_t$4) and the calibeating property (Foster et al., 2022).

Continuous Calibration via Fractional Binning:

Extending to continuously parameterized bins (fractional binning), the ensemble selects, for each continuous bin, the solution of a Brouwer-type fixed-point condition to achieve deterministic continuous calibration and calibeating (Foster et al., 2022). There is no known general polynomial-time solution for these fixed-point computations.

5. Ensemble Learning Connections, Practical Implementations, and Statistical Post-Processing

From the ensemble learning perspective, calibeating can be realized by:

• Averaging (Bagging): Simple mean aggregation corresponds to Bagging, which reduces variance and improves refinement under squared-error loss.
• Boosting: Nonlinear pooling via weighted combinations (e.g., AdaBoost, RealBoost) further improves refinement and discards underperforming forecasters, while retaining asymptotic calibration under proper loss links (Masnadi-Shirazi, 2017).

Post-processing methods, crucial in empirical systems, include:

• Platt scaling: Logistic extremization mapping $B_t = K_t + R_t$5 to correct for overconservatism (Alur et al., 10 Nov 2025).
• Isotonic regression: Nonparametric monotone recalibration, fit to minimize squared error subject to monotonicity (Alur et al., 10 Nov 2025).
• Ensemble blending of advanced AI forecasts with market consensus via simplex-constrained regression: $B_t = K_t + R_t$6, where $B_t = K_t + R_t$7 is optimized for minimum Brier error on held-out data, ensuring the aggregate cannot perform worse than the best constituent forecast in expectation (Alur et al., 10 Nov 2025).

6. Empirical Evaluations and Computational Complexity

Empirical evaluation using expert and AI forecasters demonstrates substantial improvement:

• On real-world benchmarks, ensemble calibeating methods (e.g., combining LLM-based forecasts with market consensus) strictly dominate stand-alone forecasts, achieving lower Brier scores and highly significant statistical improvements (Alur et al., 10 Nov 2025).
• Boosted ensembles on the Good Judgment Project reduce binary errors from 30 (best single) to 6 (RealBoost) over 88 questions, illustrating sharp refinement gains (Masnadi-Shirazi, 2017).

Computationally:

• Deterministic binning updates scale as $B_t = K_t + R_t$8 time plus $B_t = K_t + R_t$9 for joint bins per step, with memory linear in $K_t \to 0$0.
• Stochastic calibration requires solving a linear program over the finite $K_t \to 0$1-grid ($K_t \to 0$2).
• Continuously-calibrated, fixed-point methods are inherently more complex, with no general polynomial-time solution (Foster et al., 2022).

There is no traditional sample complexity guarantee, as all results hold in adversarial, non-i.i.d. settings (Foster et al., 2022).

7. Statistical Validity, Extensions, and Theoretical Implications

Calibeating algorithms guarantee that, for any sequence of outcomes and forecasts, cumulative loss does not exceed the benchmark refinement score of any forecaster by more than the optimal excess rates, in both expectation and with high probability. In the mixable loss setting, the minimax bounds are tight; for general bounded losses, regret scales as $K_t \to 0$3 (Chen et al., 23 Mar 2026).

This approach admits several extensions:

• Simultaneous calibration and calibeating for general proper scoring rules;
• Integration with existing prediction systems and markets, as in the AIA Forecaster’s blending of LLM and market consensus predictions (Alur et al., 10 Nov 2025);
• Embedding calibeating as a post-processing stage for any ensemble, whether detectors, classifiers, or structured prediction forecasters.

The theoretical constructs unify calibration, sharpness, online learning, and ensemble theory, providing a robust framework for expert-level probabilistic prediction surpassing the limitations of any individual source.