CITRAS-FM: Tiny Zero-Shot Forecasting Model

• CITRAS-FM is a tiny zero-shot forecasting model that supports univariate and multivariate time series with both observed and known covariates.
• It introduces Shifted Attention to align future-known covariates with forecast steps and employs CovSynth for synthetic covariate generation from target-only data.
• Empirical evaluations on fev-bench demonstrate state-of-the-art zero-shot accuracy with sub-0.1s CPU inference, highlighting its efficiency for real-time deployment.

Searching arXiv for the cited papers and closely related context papers to ensure the article is grounded in current literature. CITRAS-FM is a tiny time series foundation model for zero-shot forecasting that supports univariate targets, multivariate targets, observed covariates, and known future covariates within a single pretrained system. It is built as a patch-based, decoder-only Transformer with 7.2M parameters and is designed for sub-0.1s CPU inference, positioning it as a real-time deployable TSFM rather than a large offline forecasting model. Its two defining contributions are Shifted Attention, which temporally aligns known covariates with the forecast step they inform, and CovSynth, which synthesizes informative covariates during pretraining from target-only corpora through STL-based decomposition. On fev-bench, which comprises 100 forecasting tasks, it is reported to achieve state-of-the-art zero-shot accuracy among TSFMs with fewer than 10M parameters, with particularly strong results on covariate-informed and multivariate settings (Yamaguchi et al., 9 Jun 2026).

1. Definition and forecasting setting

CITRAS-FM addresses the conditional forecasting problem in which targets may be accompanied by two distinct covariate classes: observed covariates, available only up to the forecast origin, and known covariates, available through the forecast horizon. The target series are denoted by $$\mathbf{X}_{1:T}^{\tau,:} \in \mathbb{R}^{T \times C_\tau}$$, observed covariates by $$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$, and known covariates by $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$. The forecasting objective is to model

$$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$

This formulation is tailored to practical exogenous-input regimes in which calendar variables, planned interventions, and external forecasts are available beyond the present time. The known covariates explicitly include deterministic calendar features such as day-of-week and holidays, planned interventions such as promotions and campaigns, and external forecasts such as weather, grid load, and wind generation. In the zero-shot regime, the model is pretrained once and then applied directly to unseen series and tasks without task-specific fine-tuning; inference requires a context window of targets, optional past observed covariates, and known covariates extending into the future horizon (Yamaguchi et al., 9 Jun 2026).

The model’s scope is broader than conventional small TSFMs that are primarily univariate or do not explicitly incorporate future-known covariates. This suggests that CITRAS-FM is designed around a deployment setting in which exogenous structure is not peripheral but central to forecast formation.

2. Architectural organization

CITRAS-FM is a patch-based, decoder-only Transformer. Time series are segmented into non-overlapping patches of length $P=24$, with left or right zero-padding applied when $T$ or $T+S$ is not divisible by $P$. Each patch is then causally scaled so that normalization uses only past and current patches, preserving causal validity under non-stationarity. The patch and its binary padding mask are concatenated and embedded by a shared residual MLP into a token of dimension $D=256$.

The same embedding mechanism is shared across target variables, observed covariates, and known covariates. This parameter sharing is what allows the architecture to vary the number of targets and covariates simply by changing $C_\tau$, $$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$0, and $$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$1. The temporal backbone applies causal multi-head attention independently per variable, with RoPE for temporal order, pre-layer normalization, and SwiGLU FFNs. Temporal dynamics are therefore modeled before cross-variate interaction, rather than through a single flattened spatiotemporal attention block (Yamaguchi et al., 9 Jun 2026).

Component	Specification
Patch size	$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$2
Model dimension	$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$3
Attention heads	8
Cross-time attention depth	$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$4
Cross-variate attention depth	$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$5
Repeated block count	$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$6
Total parameters	7.2M

The output head predicts the next target patch from each target token, followed by causal rescaling to reverse the earlier causal scaling step. The shared projection network maps $$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$7 to $$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$8 and produces nine quantiles per time step,

$$\mathbf{X}_{1:T}^{o,:} \in \mathbb{R}^{T \times C_o}$$9

CITRAS-FM is therefore a probabilistic forecaster rather than a point forecaster, and its predictive distribution is represented through multi-quantile outputs rather than a parametric likelihood.

The architecture descends from the supervised CITRAS model, which also used a patch-based decoder-only Transformer for covariate-informed forecasting. CITRAS introduced KV Shift and Attention Score Smoothing for supervised settings; CITRAS-FM extends the model family into the foundation-model, zero-shot regime and replaces KV Shift with the simpler Shifted Attention mechanism (Yamaguchi et al., 31 Mar 2025).

3. Shifted Attention and covariate alignment

The central architectural innovation in CITRAS-FM is Shifted Attention within the cross-variate module. Its purpose is to align known covariates with the forecast step they inform. At patch step $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$0, the target representation is intended to predict the next target patch, so using only the covariate token at the same patch index would misalign future-known information. CITRAS-FM instead constructs the key-value set as

$$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$1

Operationally, the known-covariate token sequence is shifted left by one index and the first token is dropped. The target query at patch $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$2 attends to target tokens at $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$3, observed-covariate tokens at $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$4, and known-covariate tokens at $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$5. In standard attention notation,

$$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$6

with $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$7 and both $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$8 and $$\mathbf{X}_{1:T+S}^{\kappa,:} \in \mathbb{R}^{(T+S) \times C_\kappa}$$9 derived from the temporally shifted $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$0.

This mechanism is explicitly motivated by the need for the target at patch step $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$1 to access the covariate state relevant to the next patch, rather than only the contemporaneous patch. The paper states that standard cross-variate attention aligned at the same patch index cannot directly expose one-step-ahead covariate information. Shifted Attention is presented as a simpler and more general replacement for the more complex KV Shift mechanism used in supervised CITRAS (Yamaguchi et al., 9 Jun 2026).

A key technical point is that dropping the first known-covariate token is described as safe because $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$2 already summarizes all prior patches through the cross-time attention stack. This suggests that the model relies on temporal summarization within each variable before cross-variate fusion, allowing shifted covariate access without violating autoregressivity or introducing future-target leakage.

The practical significance of Shifted Attention is clearest in inference with future-known inputs. The model can use covariates at horizon step $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$3 when forecasting targets at that same step, even though internal processing occurs in patch units. This is the mechanism that differentiates CITRAS-FM from small TSFMs that either ignore future-known covariates or cannot align them with the prediction step.

4. CovSynth and covariate-aware pretraining

CITRAS-FM is pretrained to be covariate-aware despite the scarcity of covariate-rich large-scale corpora. The solution is CovSynth, a synthetic covariate generation method applied to target-only series. CovSynth begins with STL decomposition,

$$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$4

where $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$5 is the trend component, $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$6 the seasonal component, and $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$7 the residual component. The residual is treated as the locus of hard-to-predict variability that plausibly reflects exogenous effects (Yamaguchi et al., 9 Jun 2026).

CovSynth constructs three synthetic covariate families. The event covariate is a binary indicator for unusually high residual periods, using a threshold $$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$8 and

$$P\Big(\mathbf{X}_{T+1:T+S}^{\tau,:}\,\Big|\, \mathbf{X}_{1:T}^{\tau,:},\, \mathbf{X}_{1:T}^{o,:},\, \mathbf{X}_{1:T+S}^{\kappa,:} \Big).$$9

The long-term covariate combines trend and residual,

$P=24$0

and the periodic covariate combines seasonal structure and residual,

$P=24$1

The paper describes thresholds, mixing weights, and noise magnitudes as randomly sampled during training to produce varied covariate realizations.

The stated objective is twofold: the synthetic covariates should be informative about difficult target variability, and they should not be trivial copies of the target. This is important because most large pretraining corpora cited in the paper, including TSMixup and Gift-Eval pretraining data, are target-only. CovSynth is therefore a data-generation mechanism that allows covariate-aware representation learning in the absence of native covariate corpora.

The pretraining mix uses three datasets: TSMixup with approximately 11B time points, a Cauker-generated synthetic multivariate dataset with about 4B time points and up to 15 variables, and a Gift-Eval pretraining subset with about 19B time points. CovSynth is applied to the Gift-Eval subset, where up to five observed covariates and five known covariates are synthesized per sampled target series. At each training step, batches sample target series from the three datasets with equal probability. This mixture of real univariate data, synthetic multivariate data, and target-only data augmented with synthetic covariates is intended to expose the model to diverse target–covariate configurations (Yamaguchi et al., 9 Jun 2026).

5. Training objective and inference regime

CITRAS-FM is trained in a next-patch prediction scheme using quantile loss rather than MSE. For each quantile level $P=24$2, the pinball loss is

$P=24$3

The training loss aggregates over quantiles $P=24$4, time steps, variables, and patches. The model is trained autoregressively: for each patch it predicts the next patch, and all predicted patches in a sequence contribute to the objective (Yamaguchi et al., 9 Jun 2026).

The pretraining configuration is explicitly constrained for small-model efficiency. The maximum input length is 1032 time steps, the batch size is 256, and training runs for 500,000 steps with AdamW using learning rate $P=24$5, weight decay $P=24$6, $P=24$7, and $P=24$8. The learning-rate schedule is a 10,000-step linear warmup followed by cosine annealing, and the reported hardware is a single NVIDIA V100 32GB GPU.

A distinction is drawn between pretraining and inference. During pretraining, teacher forcing is used on ground-truth patches and future target patches are masked to preserve causality. During inference, the model predicts one patch at a time; for horizons exceeding the patch size, predicted patches may be fed back autoregressively. Known covariates across the full horizon are provided up front and are repeatedly exploited through Shifted Attention.

This training design differs materially from the supervised CITRAS objective, which used an L2 loss for next-patch point forecasting in its original formulation (Yamaguchi et al., 31 Mar 2025). The shift from MSE to multi-quantile training marks CITRAS-FM as a probabilistic foundation model rather than a deterministic supervised forecaster.

6. Empirical performance, efficiency, and position in the literature

The main empirical evaluation is on fev-bench, a 100-task zero-shot benchmark partitioned into fev-all, fev-cov, fev-multi, and fev-uni. Performance is measured by Scaled Quantile Loss relative to a SeasonalNaive baseline, with skill defined as

$P=24$9

Higher skill is better (Yamaguchi et al., 9 Jun 2026).

Among models under 10M parameters, CITRAS-FM reports 41.2% on fev-all, 39.0% on fev-cov, 54.2% on fev-multi, and 31.3% on fev-uni. These values exceed those reported for KAIROS$T$0 and Chronos-Bolt$T$1, and the paper states that YINGLONG$T$2 and TinyTimeMixer perform worse, especially on fev-uni. On fev-cov, CITRAS-FM is also described as competitive with or superior to several much larger TSFMs, including TimesFM-2.5, COSMIC, Moirai-2.0, and TiRex, while Chronos-2 and TabPFN-TS score higher but with much higher inference cost.

Model	fev-all	fev-cov
CITRAS-FM (7.2M)	41.2%	39.0%
KAIROS$T$3 (9.9M)	37.7%	35.4%
Chronos-Bolt$T$4 (8.7M)	35.9%	32.9%

The ablation results attribute this performance to both major innovations. Removing Shifted Attention reduces fev-cov from 39.0 to 36.4, while removing CovSynth reduces fev-cov from 39.0 to 37.2. The paper interprets these drops as evidence that temporal alignment of known covariates and covariate-aware pretraining both materially contribute to covariate-informed zero-shot performance.

A qualitative example is reported on EPF-NP, a day-ahead hourly electricity price task with known covariates consisting of public grid-load forecast and wind-power forecast. CITRAS-FM is said to capture price declines as grid load decreases and wind generation increases, and this behavior emerges in zero-shot use without task-specific tuning. This suggests that the model’s covariate alignment mechanism is not merely architectural but behaviorally visible in forecast trajectories.

Efficiency is evaluated on the Application dataset composed of Stan’s Robot Shop e-commerce metrics. In that setup the targets are four business KPIs, the known covariates are 35 IT event metrics, the sampling frequency is every 10 seconds, the horizon is 24 steps, and the CPU is an Intel Core i5-14400F. CITRAS-FM reports SQL of 1.09% and inference time of 0.05 s per window. Chronos-2 attains lower SQL at 0.76% but requires 1.79 s, and TabPFN-TS reports 1.38% SQL with 46.52 s inference time. The paper states that among models with sub-0.1s CPU latency, CITRAS-FM is the only one that fully leverages future-known covariates.

Relative to prior work, CITRAS-FM extends the CITRAS family from supervised forecasting into a foundation-model setting. CITRAS was introduced as a decoder-only Transformer for covariate-informed forecasting with KV Shift and Attention Score Smoothing (Yamaguchi et al., 31 Mar 2025). CITRAS-FM preserves the patch-based decoder-only design and the emphasis on target–covariate interaction, but reorients the system around zero-shot pretraining, probabilistic forecasting, real-time CPU deployment, and synthetic covariate generation. The paper positions Chronos-2 as the only existing TSFM with the same full support for univariate, multivariate, observed-covariate, and known-covariate zero-shot use, but at much higher computational cost (Yamaguchi et al., 9 Jun 2026).

The principal limitations stated for CITRAS-FM are also tightly coupled to its design. Exploiting Shifted Attention requires future-known covariates; if only past covariates are available, the model still functions but cannot use its core alignment advantage. Zero-shot behavior may degrade under extreme domain shifts, and long horizons beyond the patch size require autoregressive generation that can accumulate error. CovSynth is based on STL decomposition, so very irregular or non-seasonal series may be less compatible with its synthetic-covariate assumptions. The future directions indicated in the paper include richer covariate modeling, more realistic synthetic covariates, lightweight domain adaptation, and improved long-horizon forecasting while maintaining the small-model footprint (Yamaguchi et al., 9 Jun 2026).