Online conformal prediction with decaying step sizes (2402.01139v2)

Abstract: We introduce a method for online conformal prediction with decaying step sizes. Like previous methods, ours possesses a retrospective guarantee of coverage for arbitrary sequences. However, unlike previous methods, we can simultaneously estimate a population quantile when it exists. Our theory and experiments indicate substantially improved practical properties: in particular, when the distribution is stable, the coverage is close to the desired level for every time point, not just on average over the observed sequence.

• The paper introduces a novel online conformal prediction method with decaying step sizes that refines prediction thresholds at each time point.
• Methodology offers dual analytical guarantees, ensuring stability under both adversarial and I.I.D. data scenarios.
• Empirical tests on Elec2 and Imagenet datasets confirm superior performance over fixed step size methods in maintaining target coverage.

Introduction to the Methodology

A novel method for online conformal prediction with decaying step sizes has been introduced, presenting a significant advancement in the field of online uncertainty quantification for time-series forecasting. This approach stands apart by simultaneously offering retrospective coverage guarantees for arbitrary data sequences and facilitating the estimation of population quantiles under stable distributions. The proposed decaying step sizes contribute to more finely adjusted coverage levels for each time point, moving beyond the average coverage across an observed sequence.

Theoretical Guarantees and Comparisons

Analytical results assert dual guarantees for this method: a worst-case guarantee ensures the convergence of historical miscoverage rates to a predefined error rate even with non-identically distributed data points, while a best-case guarantee entails optimal prediction set convergence for independent and identically distributed (I.I.D.) sequences. These guarantees are not mutual in preceding models, emphasizing the unique positioning of the introduced algorithm within the existing literature spectrum. The ability to adjust to both adversarial and I.I.D. scenarios is grounded in the decaying step sizes, which are tailored through a rigorous mathematical formulation.

Algorithmic Innovations and Conformal Prediction

The methodological innovations lie both in the nuanced updates to the prediction threshold and within the underlying quantile loss function. Unlike fixed step sizes previously examined in online conformal prediction, the unique mathematical framework allows for variable step sizes, adding resilience to distributional shifts and facilitating swift recovery of coverage accuracy post any shifts. At its core, the algorithm adopts principles from conformal prediction, a domain that has garnered attention over the last century due to its broad applicability across various fields such as medicine, robotics, finance, and epidemiology.

Empirical Validation and Performance

The methodology has been verified through empirical assessments on the Elec2 dataset and Imagenet data, illuminating the practical efficacy of the proposed method. Specifically, the decaying step size procedure demonstrated superior stabilization of coverage and threshold values compared to fixed step sizes. Despite the inherently oscillatory behavior of constant step size methods, the decaying step size managed to retain coverage close to the desired confidence level, confirming its practical superiority when faced with both exchangeable data and sequences manifesting considerable distributional changes.

Final Thoughts and Future Directions

The paper presents a promising step toward bridging the gap between online conformal prediction and online learning. The use of decaying step sizes, a common practice in online gradient descent and online learning, reflects a thoughtful integration of robust optimization techniques into the field of online uncertainty quantification. Looking ahead, extending this work to tackle online risk control, accommodate non-I.I.D. time-series, and explore the nexus between online learning strategies and online conformal prediction presents a compelling trajectory for future research.