Into the ORBIT for Time Series: Training Regimes for Foundation Models

Abstract: Time series foundation models (TSFMs) have advanced primarily through architectural innovation, while training regimes for large-scale heterogeneous corpora remain under-explored. As a result, pre-training distributions are often poorly controlled with respect to domain imbalance, context requirements, prediction horizons, and missingness. We introduce ORBIT (Omni-Range Bootstrap Incremental Training), a training paradigm that makes this distribution explicit and controllable. ORBIT combines Bootstrap Multi-Level Sampling, which controls dataset exposure and samples records, target variables, context windows, and prediction horizons, with Omni-Range Incremental Training, which varies context lengths and prediction horizons throughout a single training stage. Under ORBIT, we train Falcon-2.0, a simple univariate encoder-only Transformer with missingness-aware triple-channel patch tokenization and parallel patch prediction. We further introduce Rank-Guided Cross-Depth Alignment, a training objective that uses late-layer representations as stop-gradient teachers for shallow layers without additional inference cost. Evaluations on GIFT-Eval and fev-bench demonstrate strong zero-shot forecasting performance across diverse domains and frequencies.

• The paper introduces ORBIT, combining domain-aware hierarchical sampling with single-stage training across varied contexts and forecasting horizons to control the effective pre-training distribution.
• The 585M-parameter encoder achieves the best GIFT-Eval point-forecast MASE of 0.6684 and a leading fev-bench WQL of 0.4842, while nearing TimesFM-2.5 on aggregate MASE.
• Ablations show sampling strategy matters more than most architectural choices, with sliding-window enumeration worsening GIFT-Eval MASE by 11.7% and CRPS by 13.6%, although missing covariate support limits performance on related tasks.

Motivation and positioning

Time series foundation models (TSFMs) have advanced largely through architectural innovation—group attention, flow matching, serial-token prediction, mixture-of-experts scaling—while the training regimes that govern data exposure over heterogeneous corpora have received comparatively little attention. This paper argues that a TSFM's capability is determined not by nominal corpus size but by the effective pre-training distribution: which datasets, records, variables, context windows, and horizons are actually sampled, and how missingness is handled throughout optimization. The authors identify four coupled axes along which this distribution is typically poorly controlled: cross-domain imbalance, frequency-dependent context requirements, variable prediction horizons, and missing observations.

The paper's central claim is deliberately strong: carefully designed training regimes can unlock strong forecasting capability without excessive architectural complexity. To substantiate this, the authors introduce ORBIT (Omni-Range Bootstrap Incremental Training) and train a deliberately simple 585M-parameter encoder-only Transformer under it, isolating the contribution of training-distribution design from architectural scaling.

The ORBIT paradigm

ORBIT separates the construction of forecasting examples from their consumption during optimization through two components.

Bootstrap Multi-Level Sampling operates hierarchically. At the corpus level, prescribed domain-aware dataset weights are converted into an ordered global training stream via a low-discrepancy greedy blending rule, which keeps cumulative dataset composition close to its target after every assignment rather than only in expectation. Within each retained dataset, Bootstrap Stochastic Sampling constructs an offline sample index in which each entry is a five-tuple $(r_m, v_m, s_m, e_m, p_m)$: record, target variable, context start and end, and prediction horizon. Four stochastic levels select these with equal probability among feasible options—records are not weighted by length or variable count, so exposure does not scale with the number of extractable windows. Context lengths range from one patch to $\min(C, e_m)$ and horizons from $P$ to $\min(T_{\max}, L_{r_m}-e_m)$, allowing short- and long-range targets to coexist. The index is built once, cached, and reproducible; loading occurs on demand from memory-mapped storage.

Omni-Range Incremental Training consumes this stream within a single step-based run. Contexts are left-padded and targets right-padded to mini-batch maxima, with attention and loss masks excluding unsupported positions. Because context and horizon ranges are interleaved throughout optimization, no separate context-extension stages or horizon-specific schedules are needed—an explicit contrast with multi-stage pipelines such as Timer-S1 and Chronos-2.

A comparison table of existing TSFMs shows that prior pipelines rely on uniform, sequence-count-aware, or window-count-aware dataset weighting coupled to sequential traversal or sliding-window enumeration, all of which skew exposure toward high-volume sources. ORBIT's domain-aware weighting is presented as the distinguishing feature.

Model architecture

The backbone follows the univariate encoder formulation of Chronos-2 but processes target variables independently without group attention. Key elements include missingness-aware reversible instance normalization computed only over observed values (with an arcsinh transform), triple-channel patch tokenization concatenating temporal, transformed-value, and observation-indicator features, a learnable REG token, future query patches, and direct multi-patch quantile prediction over 21 quantile levels via pinball loss. The encoder uses Pre-RMSNorm, RoPE, output-gated self-attention, and SwiGLU feed-forward layers. The default configuration has 32 blocks, $d=1024$, patch size $P=16$, maximum context $C=8192$ (512 patches), and per-stage capacity $T_{\max}=96$. Long horizons use multi-stage autoregressive prediction: normalization statistics are fixed once, the median forecast is appended recursively across stages, and scale recovery happens only after all stages.

Rank-Guided Cross-Depth Alignment is a training-only auxiliary objective motivated by the "flow of ranks" phenomenon, whereby representation rank increases across Transformer depth. A shallow block ($\ell_{\mathrm{sh}}=1$) is aligned to a deep block ($\ell_{\mathrm{dp}}=31$) through a stop-gradient, token-wise cosine loss weighted at $\min(C, e_m)$0. The paper derives a conditional spectral-transfer result: because unit-norm rows make the cosine objective equivalent to a Frobenius discrepancy, Weyl-type perturbation bounds imply that if the deep spectrum satisfies an explicit separation condition around its $\min(C, e_m)$1-th singular value, sufficiently small alignment error guarantees the shallow representation retains at least $\min(C, e_m)$2 non-negligible modes. A companion bound limits deviation in stable rank. The authors are careful to note these are conditional results—alignment does not create rank unconditionally—and that spectral breadth is a diagnostic, not itself predictive information. Stop-gradient directionality is essential: without it, the loss could be minimized by narrowing the deep representation instead.

Pre-training setup

Training uses AdamW for one million steps on NVIDIA B200 clusters under Megatron-LM with data parallelism, BF16 mixed precision, batch size 64, peak learning rate $\min(C, e_m)$3 with cosine decay. The corpus spans seven domains assembled from the GIFT-Eval pre-training split, the Chronos collection, and the Quito corpus—70 real-world datasets totaling tens of billions of observations—augmented by TSMixup and KernelSynth synthetic procedures. Data leakage prevention follows GIFT-Eval principles, with pre-training strictly separated from evaluation benchmarks.

Benchmark results

On GIFT-Eval, the model achieves the lowest Seasonal-Naive-normalized MASE among 29 pretrained models (0.6684) and the best mean MASE rank (7.81), improving over STRIDE + Timer-S1 by 0.9%. On fev-bench, its aggregate MASE of 0.6459 is within 0.3% of TimesFM-2.5 while achieving the best aggregate WQL (0.4842) among all models—the only model completing all 100 tasks with top probabilistic calibration. The fine-grained analysis exposes two clear boundaries:

Setting	Result
GIFT-Eval point forecast	Best overall (MASE 0.6684)
GIFT-Eval CRPS	Seventh (0.4843); STRIDE + Chronos-2 leads at 0.4544
fev-bench without covariates (54 tasks)	Beats Chronos-2 (MASE 0.642 vs. 0.663)
fev-bench with covariates (46 tasks)	Loses to Chronos-2 (MASE 0.652 vs. 0.621)

The covariate reversal is attributed to the channel-independent autoregressive interface, which cannot ingest known-future features—a candidly acknowledged architectural limitation rather than a training-regime failure. Domain-level results show leadership in four of seven GIFT-Eval domains (largest margin in Nature, a 10.2% reduction), with Econ/Fin remaining a weakness where Toto-2.0-2.5B dominates.

Scaling studies show consistent gains: extending training from 100k to one million steps reduces MASE by 10.6% (GIFT-Eval) and 12.1% (fev-bench), with best scores at the final checkpoint, though flattening loss suggests diminishing returns. Scaling capacity from 75M to 585M parameters improves all four metrics monotonically (e.g., fev-bench MASE falls 4.2%).

Ablations

The ablation results carry the paper's strongest quantitative support. Sampling-rule choice dominates architectural choices: replacing Bootstrap Stochastic Sampling with sliding-window enumeration over the same feasible ranges degrades GIFT-Eval MASE/CRPS by 11.7%/13.6% and fev-bench MASE/WQL by 5.4%/6.5%. Notably, even the best sliding-window variant (fixed context and horizon) remains worse than bootstrap sampling on all metrics, indicating that stochastic selection across records, variables, and temporal positions contributes diversity beyond length variation alone. Among length-sampling choices, horizon sampling matters more than context sampling (joint sampling vs. fixed-horizon yields 6.1–7.7% improvements). Architecturally, Parallel Patch Prediction is the largest contributor (8.0–8.7% degradation when removed), while triple-channel tokenization, the residual SwiGLU projection, and output gating contribute smaller but consistent gains.

Limitations and open questions

Several limitations are conceded directly. The multivariate CRPS gap against Toto-2.0-2.5B (0.476 vs. 0.446 on multivariate tasks; 0.493 vs. 0.461 at long horizons) and the overall CRPS deficit to STRIDE + Chronos-2 indicate that point-forecast strength does not fully transfer to probabilistic calibration. The inability to condition on known-future covariates reverses the performance ordering on nearly half of fev-bench tasks, defining a concrete interface question the paper leaves open. The spectral-transfer guarantee for cross-depth alignment holds only under an explicit separation condition and may be loose when the deep spectrum is highly concentrated; whether the alignment objective improves representations beyond what the forecasting loss alone achieves is analyzed qualitatively rather than isolated by ablation. Finally, the stable-rank bound's tightness and the sensitivity of results to the specific domain-aware weights are not systematically explored.

Conclusion

This paper reframes TSFM development around control of the effective pre-training distribution. ORBIT's combination of hierarchical stochastic sampling and single-stage omni-range consumption, applied to a deliberately plain encoder-only backbone, produces state-of-the-art zero-shot point forecasting on GIFT-Eval and jointly leading accuracy-calibration results on fev-bench. The ablation evidence—that sampling strategy contributes larger gains than any single architectural component evaluated—supports the paper's thesis that training-regime design is a first-order factor in building generalizable time series foundation models, while the covariate-conditioning gap and residual probabilistic weaknesses mark the boundaries of what the current formulation achieves.