Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series

Abstract: Conformal prediction methods enjoy strong theoretical and empirical predictive inference performance, provided the data is exchangeable, and predictors are trained in a memoryless fashion. However, these assumptions and constraints are impractical in many real-data settings, such as time series (where temporal dependence violates exchangeability, and where memoryless predictors will inevitably have poor predictive accuracy). Recent work shows that the split conformal prediction method is robust to these issues of memory-based predictors and deviations from exchangeability that are common features of time-series data. However, since using sample splitting can lead to lower accuracy, this motivates asking whether other predictive inference methods (that do not rely on data splitting) could also be reliably used in the time series setting. In this work, we show that the vanilla leave-one-out jackknife can suffer an arbitrary loss of coverage even in canonical time series models with mild temporal dependence. As a remedy, we propose a careful modification tailored to such settings, which we term the \emph{leave-a-window-out} (LWO) method, and show that it can achieve valid coverage provided that the model-fitting procedure satisfies mild stability properties. Our proofs are based on quantifying the degree to which the data departs from \emph{cyclic exchangeability}, and we introduce new coefficients to measure the extent of this departure. Experiments on time series data demonstrate that our LWO method often enjoys valid coverage when the vanilla jackknife fails to cover, while producing much narrower intervals than split conformal prediction.

• The paper introduces a structural modification to the jackknife, termed Leave-a-Window-Out, to correct coverage issues in time series forecasting.
• It recalibrates predictions by omitting a local window of consecutive points, thus mirroring the test-time absence of future data.
• Empirical evaluations reveal that LWO yields tighter prediction intervals and maintains nominal coverage compared to split CP and vanilla jackknife.

Leave-a-Window-Out Jackknife: Predictive Inference for Time Series

Introduction

The paper "Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series" (2605.30292) addresses shortcomings in classical distribution-free predictive inference methods for time series. Traditional conformal prediction (CP) and jackknife-based methods rely on exchangeability assumptions, which are violated in temporally dependent data. Although split CP exhibits empirical robustness to such violations, it suffers from statistical inefficiency due to sample splitting. The jackknife leverages the full data set but is shown here to severely lose coverage under time series dependence. The authors introduce the Leave-a-Window-Out (LWO) method, a structural modification to jackknife, and theoretically and empirically demonstrate its capacity to recover coverage under mild stability, achieving more efficient prediction sets than split CP in the presence of temporal dependence.

Problem Formulation and Failure Modes

Classical distribution-free prediction intervals, including CP and jackknife, guarantee coverage for exchangeable data and memoryless predictors. Time series and practical forecasting tasks violate both assumptions: observations are dependent, and predictive models utilize historical data for accuracy.

The paper mathematically formalizes the jackknife's failure using canonical time series—such as multidimensional MA(1) processes—and exposes a fundamental asymmetry: the jackknife's leave-one-out calibration points benefit from information on both sides (past and future) of the left-out data, whereas the actual test point only uses the past. Under even mild temporal dependence, jackknife calibration scores significantly underestimate test error, yielding prediction regions with arbitrarily low coverage as dimension increases.

Figure 1: Empirical coverage in a multidimensional MA(1) process with 2-NN base predictor shows severe jackknife undercoverage, while split CP and LWO maintain nominal $90\%$ coverage; LWO yields much smaller prediction region radii than split CP.

The Leave-a-Window-Out Method

To address this calibration mismatch, the LWO method systematically leaves out a local window of $\tau$ points following each proxy test point during jackknife-style calibration. This window removal ensures that, for each training data proxy test point, the model's prediction mechanism mirrors the information structure at the actual test time (where future observations are unavailable). The dependence range parameter $\tau$ is a hyperparameter informed by autocorrelation diagnostics or domain knowledge.

The general framework of LWO handles memory-based predictors, exogenous covariates, and arbitrary conformity scores, using a structural masking operation to simulate the test-time information restriction. The paper provides formal algorithmic detail for LWO, including a notation for training procedure, lifted covariates, and masking via dummy points. The method's typical computational cost is similar to jackknife, and it reduces to jackknife when $\tau=0$.

Theoretical Guarantees and Stability Conditions

To analyze LWO's validity, the authors introduce new conceptual tools. Coverage loss is bounded under mild out-of-sample (OOS) stability conditions: the method is stable if predictions do not change excessively when leaving out random training blocks. The dependence structure is quantified via the cyclic embedding coefficient, a measure of how well the time series can be embedded into a cyclically exchangeable sequence. The theory provides finite-sample bounds for LWO coverage:

• If the process is (approximately) cyclically exchangeable with lengthening $\tau$, coverage loss is bounded by $\rho_{\tau}(Z)$ and OOS stability parameters.
• For approximately Markovian or mixing sequences, the cyclic embedding coefficient is related to classical mixing coefficients $\beta(\tau)$ and $\beta^*(\tau)$.
• LWO achieves approximate coverage for fast-mixing Markov chains and general mixing processes, provided the stability condition holds. Even when Markovianity is absent, sufficient mixing and stability guarantee coverage.
• Theoretical results are derived using novel switch coefficients, TV metrics, and quantile stability lemmas.

Empirical Evaluation

Simulations confirm theoretical findings: LWO achieves valid coverage in synthetic multidimensional MA(1) processes and offers substantial efficiency gains over split CP.

Figure 2: LWO displays empirical coverage near nominal levels and tighter prediction region radii across five base predictors in MA(1) process; jackknife undercovers especially for Ridge, KNN, and Kernel Regression.

Experiments on real-world benchmark datasets (Traffic, Solar Energy) further substantiate LWO's robustness. LWO consistently maintains coverage in the presence of temporal dependence, with much narrower prediction intervals than split CP. Vanilla jackknife severely undercovers for unstable predictors or in certain regimes.

Figure 3: LWO preserves nominal coverage and smaller prediction sets on Traffic and Solar Energy datasets compared to split CP, while jackknife frequently undercovers—especially for KNN and DT predictors.

The supplementary experiments in the appendix show the same undercoverage behaviors for jackknife in challenging regime switching processes (sticky Markov chains), reinforcing the importance of structural window masking in calibration.

Figure 4: LWO maintains coverage and achieves tight prediction intervals in sticky Markov-chain simulation, where jackknife fails dramatically.

Practical and Theoretical Implications

The results underscore the necessity of structural modifications in predictive inference methods for dependent data. LWO exploits local masking to recalibrate residual distributions, avoiding optimism bias inherent in jackknife proxy scores. Its theoretical guarantees rely on stability rather than distributional exchangeability, distinguishing it from split CP’s robustness mechanisms.

Practically, LWO provides practitioners with a method to achieve valid coverage in time series forecasting without incurring the statistical inefficiency of sample splitting. The framework is agnostic to the base predictor class, making it broadly applicable—from nearest neighbor algorithms to neural networks—provided predictor stability can be ensured or validated.

The cyclic embedding and switch coefficient tools introduced here provide a new pathway for analyzing prediction procedures under dependence, with potential to inform future work on cross-conformal and bootstrap-based extensions, as well as online or spatially dependent settings.

Future Directions

The authors highlight several avenues for further research:

• Developing variants of LWO that relax stability assumptions, possibly analogs to jackknife+, CV+, or cross-conformal for dependent data.
• Designing fully data-driven selection procedures for the dependence window size $\tau$ with coverage guarantees.
• Extending cyclic embedding and switch coefficient analyses to other statistical inference settings involving non-exchangeable data.

Conclusion

"Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series" rigorously demonstrates that vanilla jackknife fails under temporal dependence, and introduces LWO as a theoretically sound data-efficient alternative. The method is supported by strong empirical results and a formal dependency-aware coverage framework. It is an essential tool for uncertainty quantification in sequential prediction applications where exchangeability and sample splitting are untenable.