Online Time Series Forecasting

• Online time series forecasting is the sequential prediction of future values, adapting continuously to nonstationarity and delayed feedback.
• Techniques like ADAPT-Z use latent feature adjustments via lightweight adapters, leveraging historical gradients to improve prediction accuracy.
• Empirical results across diverse datasets demonstrate MSE reductions of 2–12%, ensuring efficient deployment in streaming, resource-constrained environments.

Online time series forecasting is the sequential prediction of future values in a time series as data arrives, necessitating real-time adaptation to shifting data distributions and, often, the challenge of delayed ground-truth feedback. The online paradigm stands in contrast to traditional batch-mode forecasting, which relies on retraining models using blocks of accumulated data. Successful online forecasting algorithms must contend with nonstationarity, computational constraints, and the requirement to achieve low dynamic regret or cumulative loss on streaming benchmarks.

1. Problem Setting and Fundamental Challenges

Online time series forecasting is defined by the sequential arrival of inputs $\{x_t\}_{t=1}^T$ (univariate or multivariate) and the associated targets $\{y_t\}_{t=1}^T$. The forecaster produces $k$-step-ahead predictions $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$ at each time $t$, often before the ground-truth $y_{t+k}$ becomes available. Two central operational challenges recur in the literature:

• Distribution Shift (Nonstationarity): The generating joint distribution $P(x, y)$ drifts due to evolving latent factors, rendering models trained on initial data progressively suboptimal if left unadapted.
• Delayed Feedback: For $k$-step forecasting, the true value $y_{t}$ corresponding to a prediction made at $t-k$ is only observable $\{y_t\}_{t=1}^T$0 timesteps later, decoupling the adaptation of model parameters from the current state of the system and increasing the variance of online updates (Huang et al., 4 Sep 2025).

These phenomena jointly render naive online gradient approaches insufficient, motivating techniques that exploit latent structure, feature adaptation, historical gradients, or explicit mechanisms for drift estimation.

2. Latent Feature Adjustment and the ADAPT-Z Framework

Prevailing deep forecasting architectures decompose the predictive model into an encoder $\{y_t\}_{t=1}^T$1 and a prediction head $\{y_t\}_{t=1}^T$2:

$\{y_t\}_{t=1}^T$3

where $\{y_t\}_{t=1}^T$4 encodes the salient dynamics (seasonality, exogenous drivers). Empirical and theoretical advances suggest that distribution shift is often predominantly driven by changes in these latent representations, rather than the parameters of the final prediction head. As a consequence, adapting the latent features is both more effective and more stable than updating all model parameters or naively fine-tuning the prediction head in the presence of concept drift (Huang et al., 4 Sep 2025).

The ADAPT-Z algorithm operationalizes this insight by integrating a lightweight adapter $\{y_t\}_{t=1}^T$5 which computes a correction $\{y_t\}_{t=1}^T$6 as a function of the current feature $\{y_t\}_{t=1}^T$7 and a historical average of past gradients:

$\{y_t\}_{t=1}^T$8

Here, $\{y_t\}_{t=1}^T$9 is defined as the mean feature gradient over a buffer of $k$0 recently observed samples with available losses, i.e., $k$1.

The online loop at each $k$2 is:

1. Encode feature $k$3.
2. Compute delta $k$4.
3. Forecast $k$5.
4. Upon sufficient feedback (i.e., $k$6), update the adapter parameters $k$7 using current and recent observed data.

When $k$8, updates to the adapter are simply delayed accordingly, preserving robustness under feedback lag (Huang et al., 4 Sep 2025).

3. Algorithmic Strategies and Online Loop

The ADAPT-Z adapter $k$9 is constructed as a compact multilayer perceptron, fusing both the current feature and the smoothed past gradients via parallel linear layers and a shared nonlinearity. Memory and compute requirements are dominated by a buffer of $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$0 (with typically $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$1) and the small adapter ($\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$2K parameters). Hyperparameters exhibiting stability across streaming benchmarks include a gradient buffer size $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$3, learning rate $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$4, and a small or halved hidden dimension relative to $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$5. Learning rates $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$6 much above $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$7 risk divergence.

The empirical loop delays all updates by $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$8 steps, at which point:

• Historical features, predictions, and targets are retrieved for a window of length $\hat y_t = \hat g(x_t; \theta_{1},..., \theta_t)$9,
• The mean squared error loss and mean gradient in $t$0-space are computed,
• The adapter is updated by vanilla gradient descent,
• Optionally, the last-layer weights of $t$1 are fine-tuned with a much smaller learning rate $t$2.

No additional regularization (momentum or weight decay) is required beyond typical small values (Huang et al., 4 Sep 2025).

4. Empirical Performance and Benchmarking

ADAPT-Z has been extensively benchmarked across 13 data sets, including ETTh1, ETTh2, ETTm1, ETTm2, PEMS03/04/07/08, Traffic, Electricity, Solar, Weather, and Exchange. Base models such as iTransformer, SOFTS, and TimesNet are adapted to the encoder-predictor decomposition required for "Z-space" modification. Forecast horizons $t$3 and look-back windows of size 96 are typical.

Performance is measured by mean squared error, with ADAPT-Z consistently outperforming standard online learning baselines:

• Delayed online gradient descent (OGD) on full model parameters,
• Feature-only OGD (fOGD),
• Adapter-based methods such as DSOF, SOLID, ADCSD, and Proceed.

Improvement is generally 2–12% MSE reduction relative to unadapted models, with, e.g., ETTm1 at $t$4 showing MSE dropping from 0.2211 (base) to 0.1937 (ADAPT-Z), a 12.4% gain (Huang et al., 4 Sep 2025).

5. Practical Considerations and Model Deployment

Operationally, ADAPT-Z is optimized for deployment in streaming data environments with strict latency and resource constraints. The per-sample runtime is a small multiple of the unadapted model's, due to the adapter's modest size and the avoidance of any replay buffer or large historical storage. The method is robust to the choice of buffer length $t$5, learning rates, and adapter hidden size. It requires only access to intermediate feature representations ($t$6) and the final loss.

If feedback is significantly delayed ($t$7), the system simply defers gradient calculations and adapter updates accordingly, ensuring correct temporal alignment of forecast, feedback, and parameter adaptation (Huang et al., 4 Sep 2025).

6. Broader Implications and Extensions

The explicit formulation in "Z-space," targeting the most critical latent representations, yields a universally applicable and computationally efficient framework for online adaptation under nonstationarity and feedback delay. ADAPT-Z's design keeps dynamic regret low and updates stable, offering superior adaptability relative to full-parameter or final-head tuning.

As a general plug-and-play wrapper, ADAPT-Z can be coupled with any encoder–head forecasting architecture, requiring only access to the intermediate latent and prediction loss. Directions for extension include:

• More sophisticated replay strategies or gradient selection schemes,
• Richer or attention-based adapter architectures,
• Meta-training to simulate realistic streaming shifts and enhance transfer (Huang et al., 4 Sep 2025).

In summary, ADAPT-Z establishes latent feature-space adaptation via persistent historical gradients as a practical and effective paradigm for online time series forecasting, achieving state-of-the-art performance and addressing the dual challenges of distribution drift and delayed feedback in a scalable manner.