Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models

Abstract: The rapid evolution of Time Series Foundation Models (TSFMs) has advanced zero-shot forecasting across diverse domains. Inspired by the current form of LLMs, future TSFMs may be offered as commercialized, closed-source API services. However, many existing online adaptation methods still rely on white-box access for parameter fine-tuning or gradient backpropagation. This paradigm mismatch raises a question: In black-box online adaptation for TSFMs, what should we learn? We answer this with an insight: the predictive errors of the base model are conditioned on both the input and output of the base model (i.e., the context of errors). To validate this insight, we propose ORCA (Online Residual Contextual Adaptation). We conduct extensive experiments across 5 state-of-the-art TSFMs and 8 datasets to demonstrate the effectiveness of our approach. Furthermore, through ablation studies, we quantitatively analyze the impact of different adapter learning hypotheses on the final adaptation performance in black-box online adaptation. Code available at https://github.com/Fifthky/ORCA.

• The paper introduces the ORCA framework, which leverages context-conditioned linear adaptation and Boltzmann routing, achieving up to 21.2% reduction in forecasting error.
• It decomposes residuals into trend and seasonality while dynamically blending predictions, ensuring robust online adaptation in a black-box setting.
• Empirical results across 8 datasets demonstrate reliable improvements in 90% of cases with minimal adaptation overhead and low hyperparameter sensitivity.

Black-Box Online Adaptation for TSFM APIs: The ORCA Framework

Introduction and Motivation

The mainstream development of Time Series Foundation Models (TSFMs) has led to powerful zero-shot forecasters trained on broad corpora, enabling strong performance across domains. However, the increasing trend of deploying TSFMs as closed-source, commercialized API endpoints imposes strict black-box constraints on adaptivity: the user only has inference access to the model's interface and is precluded from gradient-based finetuning, internal modification, or backpropagation.

Consideration of online adaptation in this context Figure 1 reveals three adaptation paradigms: parameter finetuning (requiring white-box access), frozen white-box residual adapters (allowing gradient computation but no weight modification), and pure black-box adapters (operating entirely via the model’s observable interface). Only the third is compatible with impending TSFM API deployment models.

Figure 1: Categorization of Time Series Online Adaptation Methods. Only the black-box paradigm is viable for TSFM APIs.

The core challenge then becomes: In the black-box regime, what should be learned by the online adapter? Empirical evidence and theoretical underpinnings indicate that model errors are not merely stochastic; instead, they are conditioned on both the input context and the associated model prediction. This motivates online adaptation methodology that targets the conditional error distribution $$P(\mathbf{E}_t|\mathbf{X}_t, \mathbf{Y}_t^{\mathrm{base}})$$, modeling both what errors occur and when they occur, i.e., the context of errors rather than their marginal behavior.

The ORCA Architecture and Learning Framework

The proposed ORCA framework operationalizes this insight through a plug-and-play, lightweight black-box online adaptation pipeline. The architecture Figure 2 processes each streaming input as follows: the history window $\mathbf{X}_t$ is fed through the frozen TSFM to yield $\mathbf{Y}_t^{\mathrm{base}}$; a Linear Adapter, with strong structural bias, consumes both $\mathbf{X}_t$ and $\mathbf{Y}_t^{\mathrm{base}}$ (the "context"), decomposes the temporal residual into trend and seasonality, and outputs a correction. The Boltzmann Router dynamically integrates the base and adapted predictions according to a channel-wise confidence score derived from historical error statistics.

Figure 2: The overall architecture of ORCA with context-conditioned Linear Adapter and Boltzmann Router.

Methodologically, the design of ORCA incorporates several key components:

• Context-Conditioned Linear Adapter: Rather than regressing on past errors, the adapter performs deterministic mapping from the concatenated context $[\mathbf{X}_t, \mathbf{Y}_t^{\mathrm{base}}]$ to the expected error (i.e., conditional mean under the error distribution), leveraging the statistical property that MSE minimization targets the conditional expectation.
• Decomposition and Channel Mixing: To suppress overfitting to noise, residuals are separated into interpretable trend and seasonal components, then linearly mixed across channels.
• Boltzmann Routing: The confidence with which the adapted vs. base prediction is chosen is governed by a Boltzmann softmax over exponentially smoothed errors. This acts as a differentiable, plastic gating mechanism, yielding theoretical regret bounds analogous to the multiplicative weights update (MWU) framework and outperforming naive hard gating.
• Bayesian Predictive Loss: Continuous adaptation is implemented using a predictive-space Bayesian loss structuring online updates as a penalized regression to both new observations and a lagged prior. The gating scalar modulates the relative weighting between fitting new data and anchoring to historic predictions, implicitly regularizing abrupt shifts in high-drift scenarios.
• Cycle-Training with Decay Buffer: Experience replay is performed via a FIFO buffer with exponential decay in sampling probability, balancing adaptation plasticity against catastrophic forgetting.

Empirical Evaluation and Key Results

Adaptation Effectiveness

Extensive experiments across 8 datasets and 5 state-of-the-art TSFMs (e.g., Chronos-2, Moirai-2.0, TiRex, TimesFM-2.5, Sundial) show consistent reduction in MSE via ORCA relative to vanilla zero-shot inference, with up to 21.2% decrease in error and robust improvements in 90% of cases. Degradation, where it occurs, is strictly bounded by the Boltzmann Router and remains below 3.3%.

Figure 3: Heatmap of relative MSE drop for different adaptation methods compared to vanilla TSFM baselines. Green shows improvement, red shows degradation.

Contextual Error Modeling Hypothesis

Ablation studies dissecting adapter input hypotheses reveal that learning from past errors alone is suboptimal; conditioning on input or prediction in isolation improves performance, but concatenating both yields maximal improvements (average 6.5% MSE drop). Inclusion of past residuals does not further confer benefit, empirically confirming the conditional modeling hypothesis advocated here.

Figure 4: Impact of adapter input choices on MSE reduction; concatenated Input+Prediction is optimal.

Structural and Routing Ablations

Eliminating the Boltzmann Router notably reduces adaptation gains (average MSE drop falls from 6.51% to 5.23%). Comparison to hard routing schemes and to statistical methods (ETS, Ridge Regression) further substantiates the necessity of smooth, confidence-based blending and context-mixed adaptation, respectively.

Figure 5: Effect of structural ablations on MSE drop; the Boltzmann Router is a critical stabilizer.

Hyperparameter Robustness

Parameter sweeps for the routing temperature, EMA momentum, buffer length, and batch size reveal low sensitivity—i.e., ORCA’s performance is robust to a broad band of configuration values, granted values are set within a moderate range.

Figure 6: MSE drop relative to hyperparameter variations; adaptation is robust.

Efficiency

Measured inference and adaptation overheads are minimal relative to base TSFM inference time. On standard hardware, the additional latency incurred stays well within 100 ms per forecasted step even for large multivariate tasks, substantiating the framework's deployability.

Practical and Theoretical Implications

From a theoretical standpoint, ORCA formalizes the problem of black-box adaptation as conditional error regression, under the Bayesian online learning and expert tracking regime. The design naturally yields sublinear regret bounds and gracefully degrades to the base TSFM when adaptation is unproductive or when data exhibit adversarial shifts. In practical deployment, ORCA demonstrates the feasibility of client-side, post-hoc correction to commercial TSFM APIs, opening the door to real-time model refinement without privileged model access.

Importantly, the ablation studies offer an actionable guideline for black-box adapter construction: conditioning on both input and prediction fundamentally outperforms alternatives, marking a departure from legacy residual correction heuristics.

Online Training Pipeline

Figure 7: The online training pipeline of ORCA—snapshots are sampled into a decay-prioritized buffer, and Bayesian loss aligns outputs to ground truth and prior prediction.

Conclusion

ORCA advances the state of the art in black-box online adaptation of TSFMs, introducing a robust, low-latency, post-hoc framework for streaming scenarios where internal model access is unavailable. Empirical and theoretical analysis converge on the insight that learning the context of errors—i.e., conditioning on both input and model prediction—is essential for effective adaptation. As TSFM APIs become standard, frameworks like ORCA will constitute the practical path to adaptive, resilient time series forecasting. Limitations remain (e.g., reliance on base model competence and deterministic metrics), and future directions include handling abrupt shifts and extending the approach to probabilistic, multimodal metrics.

(2606.14222)