TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning

Abstract: Foundation models mark a profound paradigm shift in time series modeling, with task-specific models being superseded by general-purpose zero-shot models. Yet, current approaches primarily focus on forecasting, while real-world time series are often irregularly and partially observed, requiring models that can jointly forecast, impute missing values, and handle degraded sampling conditions. To address these challenges, we introduce TS-ICL, a novel probabilistic In-Context Learning encoder--regressor Transformer that unifies forecasting and imputation. TS-ICL formulates time series tasks as timestamp-aligned regression and naturally incorporates covariates by training on synthetic dependency structures generated from a novel causal data prior. Empirically, TS-ICL achieves a new state-of-the-art in imputation, while remaining competitive with leading forecasting foundation models across both univariate and covariate-aware benchmarks. It shows particularly strong performance in forecasting with partially observed look-back windows.

• The paper presents a unified model that reframes forecasting and imputation as a continuous, timestamp-aligned in-context regression problem.
• The methodology employs a Perceiver-style encoder and cross-channel attention to handle arbitrary timestamps, missingness, and covariate integration.
• TS-ICL leverages structured causal priors and synthetic data to achieve state-of-the-art imputation performance and competitive forecasting accuracy.

TS-ICL: A Flexible Time-Indexed Foundation Model for Unified Time Series Forecasting and Imputation

Introduction and Motivation

Traditional Time Series Foundation Models (TSFMs) have catalyzed a shift away from task-specific modeling approaches toward general-purpose, zero-shot predictors through pretraining on heterogeneous mixtures of real and synthetic data. However, most TSFMs focus almost exclusively on forecasting, with limited support for imputation, subpar robustness to irregular or incomplete inputs, and insufficient mechanisms for covariate-aware inference. Existing tabular foundation models, when adapted for temporal data, enable broader in-context prediction (including imputation), but lack crucial temporal inductive biases, depend on engineered time features, and suffer severe computational inefficiencies.

The TS-ICL model directly addresses these deficiencies by unifying zero-shot forecasting and imputation within a continuous-time, time-indexed, in-context learning framework predicated on flexible probabilistic modeling over arbitrary timestamped observations. It supports non-gridded, partially observed, and heterogeneously sampled series—including dense, block-missing, or highly-irregular histories—while natively integrating exogenous covariate information with a novel DAG-based structured causal prior during training.

TS-ICL Architecture and In-Context Regression Pipeline

TS-ICL re-formulates time series prediction as a timestamp-aligned in-context regression problem, implemented via a modular pipeline designed to accommodate arbitrary observation timestamps, missingness patterns, and covariate sets.

The model pipeline comprises four principal modules:

1. Time Series Encoder ($\mathcal{E}$): Uses a Perceiver-style design to encode timestamp–value pairs for both target and covariate channels, mapping each to a fixed number of learnable latent tokens via cross-attention between timestamp positional embeddings and learnable queries. This step is channel-independent and produces compact, geometry-aware representations.
2. Channel Mixer ($\mathcal{M}$): Aggregates across all variables (target and covariates) through cross-channel attention, collapsing multiple channel-specific tokens into a single covariate-aware representation. Global dependencies are modeled via self-attention amongst the aggregated latent tokens.
3. Temporal Context Query Module ($\mathcal{C}$): Enables querying of the assembled contextual latent codes at any arbitrary timestamp using sinusoidal/Fourier-based positional encodings. This cross-attention bridges discrete latent variables with the continuous time axis, supporting regular and irregular sampling seamlessly.
4. In-Context Regressor ($\mathcal{R}$): A causal Transformer taking sequences of (embedding, value) pairs from observed context, plus queries for target timestamps (with or without covariates), and producing probability quantiles for each target through in-context regression.

This architecture enables TS-ICL to accommodate missing or irregularly sampled input without preprocessing or gridding, to leverage arbitrary sets of optional covariates, and to provide a unified probabilistic output across forecasting and imputation tasks.

Figure 1: The TS-ICL pipeline, illustrating the end-to-end transformation from temporal encoding through in-context regression for a forecasting task with a covariate observed on the horizon.

Structured Causal Data Priors and Covariate Synthesis

To facilitate robust zero-shot generalization and covariate-aware inference, TS-ICL is pretrained on large mixtures of real and diverse synthetic time series, augmented by a structured DAG-based causal prior. This prior generates multivariate time series, where nodes correspond to series and edges specify dependencies instantiated through a registry of both linear and nonlinear Structural Causal Models (SCMs). Target–covariate prediction problems are dynamically sampled by selecting one node as the prediction target and others as exogenous inputs, allowing the inclusion of both informative and redundant (or even independent) covariates at training time. This design ensures that the model learns not just to exploit but to disregard non-informative covariates, supporting robust, context-sensitive exogenous integration.

Experimental Results

Zero-Shot Imputation

TS-ICL establishes new state-of-the-art performance in zero-shot imputation, outperforming task-specific models and tabular foundation models—including TabICLv2-TS and TabPFNv2.5-TS—across both univariate and covariate-aware benchmarks. On the fm-impute-bench benchmark, TS-ICL reduces NMAE and CRPS over TabICLv2-TS by 17% and 15%, respectively, and is approximately 50× faster at inference. When covariates are available, the positive margin in both metrics increases over the closest TFM baseline, and the model demonstrates robust accuracy improvements with covariate integration.

Figure 2: Performance of TS-ICL and competitors for univariate imputation across 132 tasks, as measured by NMAE and CRPS.

Zero-Shot Forecasting

On fev-bench and TIME benchmarks, TS-ICL matches or closely approaches the strongest TSFM competitors (e.g., Chronos-2, TiRex) in both MASE and CRPS metrics, and clearly outperforms tabular models and local baselines. In the known-covariate forecasting setting, TS-ICL leverages exogenous inputs to further close the gap to the leading patch-based TSFM, and achieves particularly strong performance when historical context is incomplete—an area where patch-based TSFM architectures degrade sharply.

A notable result is that under massive context missingness (e.g., 70–90% of the look-back window absent), TS-ICL exhibits substantially smaller degradation in predictive accuracy compared to Chronos-2, confirming the benefits of its time-indexed, non-patch-based formulation.

Figure 3: Task-averaged MASE and CRPS for univariate forecasting (100 tasks), demonstrating TS-ICL’s close performance to leading TSFMs and superiority over TFMs.

Figure 4: TS-ICL producing calibrated forecasts with horizon 168 and one covariate on a GFC17 task, evidencing practical covariate integration.

Implications, Limitations, and Future Directions

TS-ICL's architecture demonstrates that unifying imputation and forecasting as timestamp-aligned in-context regression enables strong zero-shot performance and practical applicability for realistic, messy time series—spanning diverse domains, window lengths, frequencies, missingness patterns, and with optional exogenous information.

Empirical claims in the paper include:

• TS-ICL yields SOTA imputation performance (17–36% better on NMAE versus best tabular baselines),
• achieves competitive SOTA forecasting results, and
• shows markedly improved robustness to massive missingness in the look-back compared to the dominant patch-based TSFM approaches.

The primary limitation is computational efficiency: although TS-ICL achieves fast inference versus tabular models, it is up to 4× slower than highly optimized patch-based models such as Chronos-2 on equivalent hardware—a direct consequence of its pointwise in-context regression strategy.

Practical and theoretical implications include (i) a strong paradigm for unified treatment of time series tasks (forecasting, imputation, potentially anomaly detection and classification); (ii) an architectural blueprint for handling heterogeneous, missing, and irregular data without preprocessing; and (iii) a demonstration that structured causal synthetic priors can be leveraged to teach models both the use and rejection of covariate information. Future extensions should focus on architectural acceleration (e.g., via caching or mixed-precision), broader synthetic data prior diversity, and exploring broader generalization to anomaly detection and zero-shot classification.

Conclusion

TS-ICL advances the field of time series foundation models by providing a unified, probabilistic, in-context learning approach capable of robustly handling forecasting and imputation—even in the presence of irregular and incomplete data, and with optional, possibly-irrelevant exogenous covariates. It establishes a new empirical benchmark for imputation, delivers competitive forecasting performance, and evidences strong architectural synergies between temporal latent encoding and in-context regression. TS-ICL's flexibility, accuracy, and robustness position it as a strong alternative to canonical patch-based TSFM pipelines, and as a viable foundational template for future universal time series modeling approaches.

---

Reference:

"TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning" (2606.05878)