CITRAS-FM: Tiny Time Series Foundation Model for Covariate-Informed Zero-Shot Forecasting

Abstract: Pretrained time series foundation models (TSFMs) have enabled zero-shot forecasting on unseen target series. However, existing TSFMs often incur high computational cost and provide limited support for diverse variable types, often failing to account for covariates that exogenously influence target variability. To address these challenges, we propose CITRAS-FM, a tiny 7M-parameter TSFM that supports univariate, multivariate, and covariate-informed zero-shot forecasting with real-time CPU inference. Built on a patch-based, decoder-only Transformer, CITRAS-FM introduces Shifted Attention into the cross-variate module to effectively exploit known covariates accessible throughout the forecast horizon. Moreover, to enable covariate-aware pretraining despite the scarcity of covariate-rich corpora, we propose CovSynth, which synthesizes realistic covariates from decomposed components of target series. Experiments on fev-bench, spanning 100 tasks across various settings, demonstrate that CITRAS-FM achieves state-of-the-art zero-shot accuracy among sub-10M TSFMs while delivering sub-0.1-second CPU inference, offering a strong balance between forecasting accuracy and real-time deployability.

• The paper introduces a 7M-parameter TSFM that efficiently performs zero-shot forecasting using a patch-based Transformer architecture.
• It employs novel Shifted Attention and a CovSynth pipeline to robustly exploit future-known covariates, enhancing forecast precision.
• Empirical results on fev benchmarks demonstrate superior accuracy and real-time CPU inference, outperforming larger models in covariate-intensive tasks.

CITRAS-FM: Compact Covariate-Informed Time Series Foundation Modeling for Zero-Shot Forecasting

Introduction and Motivation

Forecasting future values in time series is essential in domains where exogenous variables and dynamic multivariate dependencies are prevalent. The emergence of pretrained Time Series Foundation Models (TSFMs)—models trained on massive corpora for zero-shot generalization—has advanced the field, yet practical barriers remain. Most leading TSFMs are computationally intensive, limiting their deployment in latency-sensitive or resource-constrained environments. Additionally, a significant gap persists in the ability of TSFMs to handle diverse settings involving univariate, multivariate, and, critically, covariate-enriched inputs, particularly when future-known covariates (variables observable for the forecast horizon) are involved.

CITRAS-FM introduces a 7M-parameter architecture bridging this gap by enabling efficient, flexible zero-shot forecasting, even in scenarios with multiple targets and covariate signals. The model's lightweight design, real-time inference on commodity CPUs, robust covariate exploitation via architectural innovations, and covariate synthesis for pretraining fundamentally enhance the practical utility of TSFMs.

Model Architecture

CITRAS-FM leverages a patch-based, decoder-only Transformer backbone, extending the CITRAS framework to the foundation modeling regime and augmenting it for covariate awareness. The architecture is modular, composed of input projection, cross-time attention, cross-variate attention (with novel Shifted Attention), and output projection.

The initial input projection transforms raw time series into patch-wise token embeddings, preserving both local temporal structure and non-stationary characteristics via patching and causal scaling. Both targets and covariates undergo this embedding step with shared parameterization for variable-agnostic processing.

Cross-time attention models temporal dependencies within each variable using RoPE positional encoding and multi-head attention with causal masking. Pre-layer normalization and SwiGLU activations are employed for optimization stability.

Critically, the cross-variate module introduces Shifted Attention to align known covariates with the forecasted targets: at each prediction step, patches of known covariates are shifted so that each patch has direct access to next-step known covariate embeddings, supporting optimal conditioning for forecasting future values.

Figure 1: Overall structure of CITRAS-FM, illustrating the modular path from input patch projection through cross-time and cross-variate attention with Shifted Attention, concluding in probabilistic output generation.

The output module produces quantile-based probabilistic forecasts for each patch, supporting direct uncertainty estimation.

CovSynth: Covariate Synthesis for Pretraining

Robust foundation modeling in covariate-rich settings requires access to diverse and realistic covariate patterns in pretraining corpora, a resource that is typically scarce. The authors address this with CovSynth, a novel data generation pipeline synthesizing plausible covariates from target time series.

CovSynth operates by decomposing time series into trend, seasonal, and residual components via STL. From residuals, it generates three pseudo-covariate types representing event-driven, long-term, and periodic exogenous phenomena. This not only increases covariate diversity during pretraining but also ensures that covariates capture target-irreducible information, enhancing transferable generalization to real-world, covariate-informed forecasting tasks.

Figure 2: Procedure of CovSynth for creating event, long-term, and periodic pseudo-covariates from target series decompositions.

Experimental Evaluation

Evaluation on the fev-bench—a 100-task suite spanning univariate, multivariate, and covariate-informed forecasting—demonstrates that CITRAS-FM achieves the highest zero-shot forecast skill among models under 10M parameters, outperforming several models with parameter counts exceeding 20 times larger in covariate-rich settings.

Specifically:

• On the full fev-bench ("fev-all"), CITRAS-FM attains a skill score of 41.2%, surpassing all tiny TSFMs.
• In covariate-informed tasks ("fev-cov"), CITRAS-FM outperforms larger models like TimesFM-2.5 and COSMIC despite their computational disadvantage.
• Inference latency on CPU remains below 0.1 seconds per forecast window, making the model well-suited for real-time deployment.

A qualitative example in electricity price forecasting illustrates the model's ability to leverage grid load and wind power covariates for market-aware, zero-shot prediction calibration.

Figure 3: Zero-shot forecasting on the EPF-NP task, showing CITRAS-FM's accurate adaptation to market-driven shifts using covariate inputs.

In an application setting with resource constraints, CITRAS-FM is the only sub-10M model employing future-known covariates, achieving a significant reduction in quantile loss while maintaining low inference time.

Ablation studies confirm that both Shifted Attention and CovSynth contribute substantively to covariate-informed skill; their removal yields notable declines in benchmark accuracy.

Implications and Future Directions

CITRAS-FM demonstrates that targeted architectural modification (Shifted Attention) and innovative pretraining data augmentation (CovSynth) enable compact TSFMs to rival or exceed much larger models in both flexibility and forecasting precision. This raises the standard for TSFM deployability in operational settings where low-latency and support for heterogeneous variable types are mandatory.

The work implies several future research avenues:

• Extension of covariate synthesis for richer and higher-dimensional exogenous feature modeling.
• Exploration of adaptive parameter scaling for deployment across heterogeneous hardware tiers.
• Formal study of transferability and robustness under various model and covariate distributional shifts.
• Investigating interpretability of Shifted Attention in relation to covariate influence attribution.

Conclusion

CITRAS-FM establishes a new efficiency-accuracy frontier for zero-shot time series foundation models by marrying an efficient, patch-based Transformer architecture with innovations for covariate alignment and data augmentation. The strong empirical results in covariate-informed settings and real-time CPU applicability mark substantive advances in the practical deployment of TSFMs. The approach broadens the feasibility of zero-shot forecasting across a wider set of industrial and scientific domains where low-latency and flexible variable handling are decisive factors.