Rethinking Early Stopping: Refine, Then Calibrate (2501.19195v2)

Abstract: Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.

• The paper introduces a novel early stopping strategy that decouples minimizing refinement and calibration errors by focusing on refinement during training and applying post-hoc calibration.
• Their method uses a refinement-based stopping criterion estimated via TS-refinement to guide training, allowing for better performance when calibration is applied afterwards.
• Empirical results show this approach significantly reduces downstream errors, achieving up to 25% lower cross-entropy on benchmark datasets and improving loss on tabular data.

The paper "Rethinking Early Stopping: Refine, Then Calibrate" presents a novel approach to mitigating the compromises inherent in the traditional training of machine learning classifiers, particularly concerning calibration and refinement errors. The authors argue that probabilistic predictions from classifiers, which are fundamental to many applications, are often miscalibrated. These miscalibrations arise because conventional methods minimize both calibration and refinement errors at the same point during training. They suggest that this simultaneously affects the model's confidence and its ability to distinguish between classes suboptimally.

Key Contributions and Methodology

1. Calibration and Refinement Errors:
   • The paper delineates two components of proper loss: calibration error and refinement error. Calibration error assesses the model's confidence alignment with true probabilities, while refinement error evaluates the model's ability to distinguish between classes.
   • A critical observation is that these errors do not generally achieve their minima at the same epoch during training, leading to a suboptimal compromise when standard validation loss is used for early stopping.
2. Refinement-Based Early Stopping:
   • The authors propose a paradigm where refinement error is minimized during the training phase, and calibration error is addressed post-training through techniques like temperature scaling (TS).
   • Their approach includes a refinement-based stopping criteria that uses TS-refinement, an estimator for refinement error, making it possible to train models with reduced loss when post-hoc calibration is applied.
3. Theoretical Insights:
   • The authors offer a variational characterization of the decomposition of risk into calibration and refinement components, framing the calibration error as the remaining expected loss after optimal re-labeling. This re-framing allows for estimating refinement error as the minimum achievable loss via post-hoc calibration.
4. Practical Implementations and Results:
   • The proposed TS-refinement method is designed to integrate seamlessly with existing architectures, and empirical evidence is presented showing significant improvements across diverse classification tasks, particularly evident in neural networks.
   • Experiments conducted on computer vision datasets (e.g., CIFAR-10, CIFAR-100, SVHN) using ResNet and WideResNet architectures demonstrate up to 25% reduction in downstream cross-entropy error through their methodology.
   • Benchmark results for tabular data validate the robustness of their method, as TS-refinement consistently improves loss even on large and diverse datasets.
5. High Dimensional Theoretical Analysis:
   • Analyzed in a high-dimensional logistic regression setting, the paper explores the alignment between calibration and refinement errors' optimizers. They provide a Gaussian data model analysis showing that even simple settings exhibit mismatches in these optimizations.

Conclusion

This work introduces an efficient method for reducing the risk of machine learning classifiers by decoupling the training strategy for calibration and refinement errors. The authors establish a strong theoretical foundation and deliver robust empirical analyses to support their approach. They highlight potential applications beyond traditional machine learning problems, suggesting benefits in settings like fine-tuning large foundational models where long training passes can be necessary for optimization. The technique's practicality and the release of an accompanying code package make it accessible for immediate adoption in diverse machine learning workflows.