CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.

• The paper introduces CalArena, a comprehensive benchmark that standardizes the evaluation of post-hoc calibration in ML classifiers.
• It aggregates nearly 2000 experiments across diverse modalities and models, comparing parametric, non-parametric, and native multiclass calibration techniques.
• Empirical results demonstrate that smooth, regularized methods outperform binning-based and generic approaches, especially in high-dimensional scenarios.

CalArena: A Comprehensive Benchmark for Post-Hoc Calibration

Motivation and Benchmark Design

CalArena addresses persistent deficiencies in the evaluation of post-hoc calibration in ML classifiers. Miscalibration undermines confidence in probabilistic predictions, impacting downstream applications where reliability of uncertainty estimates is critical (e.g., healthcare, risk management, autonomous systems). Existing literature is fragmented by small-scale benchmarks, inconsistent metrics, and a proliferation of methods without unified implementations. CalArena introduces a rigorously standardized, large-scale benchmark comprising nearly 2000 experiments across binary, multiclass, and large-scale classification settings, spanning tabular and computer vision modalities, classical and deep architectures, including foundation models.

Benchmark construction aggregates predictions from disparate repositories such as TabRepo, TabArena, and CV datasets, covering both classical ML and recent foundation models in tabular and vision domains. Each experiment is defined by a dataset-model pair with separate validation/test predictions, enabling robust calibration evaluation. All benchmark components, data, and code are openly released, facilitating reproducibility and plug-and-play extensibility.

Calibration Methods and Evaluation Perspective

CalArena implements a wide spectrum of calibration algorithms within a unified package, supporting both binary and multiclass settings. Binary methods include parametric logistic models (Platt scaling on logits/probs, Temperature Scaling, Quadratic Scaling, Beta calibration), non-parametric schemes (Isotonic Regression, Centered Isotonic Regression [CIR], Venn-Abers, histogram/quantile binning, BBQ), splines, kernel-based estimators, and tree-based classifiers (LightGBM, XGBoost, CatBoost).

For multiclass calibration, native approaches (Temperature Scaling, Vector Scaling [VS], Matrix Scaling [MS], Dirichlet calibration, Structured Matrix Scaling [SMS], Structured Vector Scaling [SVS], ETS, kernel-based methods) are contrasted with binary methods extended via one-versus-rest (OvR). The benchmark carefully distinguishes between methods' suitability and scalability as class count increases.

Crucially, the evaluation metric departs from traditional calibration error estimators (e.g., ECE), which are sensitive to binning and lack consensus in multiclass settings. Instead, CalArena introduces Post-Hoc Improvement (PHI) in proper scoring rules, primarily the Brier score, which integrates calibration and refinement errors. PHI quantifies the net change in expected loss after calibration, rewarding calibration that preserves sharpness.

Empirical Findings

CalArena provides empirical leaderboards of averaged winrates, uncertainty intervals derived via bootstrap on datasets/experiments, and Elo scores via Bradley-Terry modeling. Key numerical results establish several robust conclusions:

• Smooth calibration functions consistently outperform binning-based alternatives across benchmarks.
• Parametric logistic models (Quadratic Scaling, Platt-logits, Beta calibration) are the top performers in binary calibration.
• For multiclass calibration, Structured Matrix Scaling (SMS) achieves dominant winrates, especially as dimensionality increases.
• OvR strategies, while competitive in low-class problems, deteriorate in high-dimensional settings; native multiclass methods are essential when $K$ grows.
• Generic ML models (gradient boosting) are not competitive as calibrators without domain-specific design adaptations.

Figure 1: Benchmark results for binary and multiclass calibration, showing method winrates and highlighting the superiority of smooth, parametric approaches in both settings.

In binary benchmarks, the difference in winrate between the best (e.g., Spline calibration, Quadratic scaling, Beta calibration) and worst performers is modest on classical tabular datasets but grows substantially in advanced model and vision benchmarks. For multiclass calibration, SMS and SVS consistently outperform others; the methodology's regularization hierarchy, avoiding over-parameterization, proves critical. The overfitting tendency of highly flexible models (e.g., MS, Dirichlet) is evidenced by their failure to surpass simpler approaches (e.g., TS) in high-class scenarios.

Analysis of Scaling with Class Dimensionality

A critical analysis investigates the scaling of OvR and native multiclass methods as class count $k$ increases. SMS provides robust performance gains even as $k$ grows, outperforming Spline calibration applied OvR beyond $k=3$. This delineates a clear operational boundary: parametric multiclass models with hierarchical regularization are necessary for calibration in high-dimensional classification.

Figure 2: Winrate of SMS compared to Spline calibration as the class count increases, evidencing the necessity of native multiclass approaches for high-dimensional settings.

Runtimes and Practical Considerations

CalArena also benchmarks runtime efficiency for all methods, normalizing compute per 1000 calibration samples per class. Non-parametric procedures (isotonic, binning) are fast for binary, but scale poorly with $K$ in multiclass. Parametric models (SMS, TS) remain tractable, further justifying their recommendation for large-scale applications.

Figure 3: Average runtime per 1000 samples for binary calibrators across all benchmarks, with smooth, parametric approaches offering competitive efficiency.

Figure 4: Average runtime per 1000 samples per class for multiclass calibrators, demonstrating the scalability of SMS and related parametric methods.

Robustness, Statistical Significance, and Practical Recommendations

Critical difference diagrams and Elo score leaderboards affirm these conclusions with statistical significance: methods such as Quadratic Scaling, SMS, and SVS form significantly superior groups. The inclusion of non-calibrated predictions as a baseline consistently reflects the risk incurred by poor calibration choices, with several methods degrading performance below baseline.

Figure 5: Elo-based ranking of calibration methods for binary and multiclass benchmarks, cementing the relative superiority of smooth, regularized models.

Implications and Future Directions

CalArena's results advance multiple practical and theoretical implications:

• Practitioners should prefer smooth, parametric calibration functions (quadratic/logistic scaling, SMS/SVS) for post-hoc calibration in both binary and multiclass settings, particularly for modern, high-dimensional classifiers.
• OvR approaches are only recommended for small $K$; high-dimensional multiclass calibration must employ native multiclass regularized models.
• Calibration-specific architectural constraints (regularization, monotonicity) are required when adapting generic ML models as calibrators.
• PHI in proper scoring rules (preferably Brier score) is a principled metric for benchmark evaluation and should supersede ECE, especially for comparative studies.

CalArena’s plug-and-play infrastructure, open data, and standardized evaluation protocols provide an actionable resource for the calibration research community. The extensibility invites ongoing contributions, promising to maintain a dynamic leaderboard and driving reproducibility standards.

Future work should expand data/modalities (e.g., generative/NLP calibration with massive class counts), incorporate implementation-tuned hyperparameter ranges, and further benchmark newly proposed methods as computational feasibility improves.

Conclusion

CalArena establishes a rigorous, large-scale, and reproducible foundation for the empirical evaluation of post-hoc calibration. The empirical evidence underscores the dominance of smooth, parametric calibration models and hierarchical regularization in both binary and multiclass contexts, particularly as dimensionality and model complexity scale. Binning-based and generic ML methods, unless specifically adapted, are not competitive.

The benchmark's methodology and infrastructure are poised to inform both future practical deployments and theoretical investigations into calibration in ML. By providing a unified, extensible platform, CalArena is central to the advancement of trustworthy probabilistic predictions in modern AI.