Toto 2.0: Scalable Forecasting Models

Updated 4 July 2026

Toto 2.0 is a family of open-weight time-series forecasting models characterized by a unified training recipe, scalable architecture, and demonstrable state-of-the-art performance.
Its innovative design replaces autoregressive patch decoding with contiguous patch masking and parallel forecasting to reduce rollover errors and lower latency.
Spanning 4M to 2.5B parameters, the model family leverages both observability and synthetic data to set new benchmarks on BOOM, GIFT-Eval, and TIME.

Toto 2.0 is a family of open-weight time-series foundation models for forecasting built around the claim that forecasting quality can improve reliably with scale under a single training recipe. The family comprises five base checkpoints spanning 4 million to 2.5 billion parameters, is released under Apache 2.0, and is reported to set new state of the art on BOOM, GIFT-Eval, and TIME. Its central significance lies less in any single checkpoint than in the accompanying demonstration that a common architecture, optimizer stack, data mixture, and u- $\mu$ P transfer pipeline can be used from small to billion-parameter regimes without the erratic scaling behavior that had characterized many earlier time-series foundation models (Khwaja et al., 19 May 2026).

1. Lineage and conceptual position

Toto 2.0 emerged from Datadog’s earlier Toto line, which had already established an observability-centered forecasting agenda using decoder-only, patch-based transformers trained on large mixtures of observability, public, and synthetic time series (Cohen et al., 2024). In its 2025 open release, Toto was framed as a 151M-parameter forecasting foundation model specialized for multivariate observability data and benchmarked on BOOM, a large-scale observability benchmark (Cohen et al., 20 May 2025).

Toto 2.0 repositions that line in two ways. First, it shifts emphasis from observability specialization alone to general scaling behavior across domains. Second, it replaces several core design choices of earlier Toto models. Earlier releases relied on autoregressive next-patch decoding and Student- $T$ mixture heads; Toto 2.0 is explicitly described as a substantial redesign centered on contiguous patch masking, parallel decoding, quantile forecasting, and a width-transfer training pipeline based on u- $\mu$ P (Khwaja et al., 19 May 2026).

This redesign also changes the evidentiary structure of the project. The earlier Toto papers argued primarily from strong zero-shot performance and observability relevance (Cohen et al., 2024, Cohen et al., 20 May 2025). Toto 2.0 instead argues that scaling itself has become a practical control knob for forecasting quality. A plausible implication is that the work aims to move time-series foundation models from an architecture-comparison phase into a recipe-stability phase analogous to other foundation-model domains.

2. Model family and backbone

All five Toto 2.0 base models share a decoder-only patched-transformer backbone with alternating attention views over time and over variates. They are trained with context length 4096, patch size 32, 32 variates per sample, and global batch size 64. The attention head dimension is fixed at $d_{\text{head}} = 64$ for all sizes (Khwaja et al., 19 May 2026).

Model	Parameters	Backbone configuration
4M	4M	$d_{\text{model}}=256,\ h=4,\ L=4$
22M	22M	$d_{\text{model}}=512,\ h=8,\ L=6$
313M	313M	$d_{\text{model}}=1024,\ h=16,\ L=24$
1B	1B	$d_{\text{model}}=1536,\ h=24,\ L=36$
2.5B	2.5B	$d_{\text{model}}=2048,\ h=32,\ L=48$

The architectural backbone inherits the broad Toto design of patched multivariate forecasting, but with several notable changes. Time-axis attention is causal; variate-axis attention is full across variables. The best configuration places variate-axis attention last in the stack. Toto 2.0 also adds PerDimScale as learned per-dimension query scaling, uses biases in attention projections but not in MLPs, excludes all-missing patches from attention, and uses no dropout (Khwaja et al., 19 May 2026).

The patch input and output projections are no longer simple linear maps. They are implemented as two-layer SiLU MLPs with residual connections. This introduces nonlinear representation learning both before and after the transformer core. Input scaling is likewise revised: Toto 2.0 applies a robust causal scaler with an $\operatorname{arcsinh}$ transform,

$T$ 0

chosen because it is linear near zero, logarithmic in the tails, and sign-preserving, which is particularly relevant for observability metrics spanning many orders of magnitude (Khwaja et al., 19 May 2026).

3. Forecasting mechanism and probabilistic head

The central architectural innovation is contiguous patch masking (CPM). Instead of forecasting one patch at a time autoregressively, Toto 2.0 predicts all masked horizon patches in one forward pass. If $T$ 1 is the set of masked patches, then

$T$ 2

where each patch carries a binary mask channel $T$ 3. During inference, the horizon is masked as

$T$ 4

This replaces the older autoregressive form

$T$ 5

which required $T$ 6 sequential transformer calls for a $T$ 7-patch horizon (Khwaja et al., 19 May 2026).

The immediate consequences are parallel forecasting, reduced rollout error accumulation, and lower latency. Toto 2.0 also supports block decoding, in which the horizon is predicted in blocks of $T$ 8 patches; after each block, decoded patches are committed via

$T$ 9

Single-pass decoding is the standard evaluation mode, while block decoding is used for long-horizon stability studies (Khwaja et al., 19 May 2026).

A second major change is the replacement of Toto 1.0’s Student- $\mu$ 0 mixture head with a quantile head predicting

$\mu$ 1

For target $\mu$ 2 and predicted quantile $\mu$ 3, the pinball loss is

$\mu$ 4

and the aggregate loss is

$\mu$ 5

The paper motivates this change by numerical instability in large-scale Student- $\mu$ 6 mixtures, especially near zero under variance normalization. At inference, the predicted quantiles are sorted to avoid crossing (Khwaja et al., 19 May 2026).

4. Training recipe, optimization, and u- $\mu$ 7P transfer

Toto 2.0 treats scale as a training-recipe problem. The optimizer is hybrid: NorMuon is used for internal matrix-shaped parameters, while AdamW is used for input/output projections, biases, and normalization parameters. The argument for NorMuon is tied to the pinball-loss gradient,

$\mu$ 8

which is effectively sign-valued and therefore poorly matched to variance-adaptive optimization at the per-parameter level (Khwaja et al., 19 May 2026).

The NorMuon update used in Toto 2.0 is

$\mu$ 9

with row-wise normalization of the orthogonalized momentum update $d_{\text{head}} = 64$ 0. The selected hyperparameters are: NorMuon learning rate $d_{\text{head}} = 64$ 1, momentum $d_{\text{head}} = 64$ 2, $d_{\text{head}} = 64$ 3, and weight decay $d_{\text{head}} = 64$ 4; AdamW learning rate $d_{\text{head}} = 64$ 5, $d_{\text{head}} = 64$ 6, and $d_{\text{head}} = 64$ 7; warmup steps 6000; gradient clipping threshold 7.0; and a warmup-stable-decay schedule with a linear 10,500-step tail (Khwaja et al., 19 May 2026).

Hyperparameter transfer across scales is handled with u- $d_{\text{head}} = 64$ 8P. The proxy model is approximately 10M parameters with $d_{\text{head}} = 64$ 9, $d_{\text{model}}=256,\ h=4,\ L=4$ 0, and $d_{\text{model}}=256,\ h=4,\ L=4$ 1, trained for 30,000 steps per trial. Search proceeds in four rounds—architecture, data mixture, optimizer, and decay schedule—using Optuna with TPE. Hidden weights are parameterized as

$d_{\text{model}}=256,\ h=4,\ L=4$ 2

with updates

$d_{\text{model}}=256,\ h=4,\ L=4$ 3

and scaling rules

$d_{\text{model}}=256,\ h=4,\ L=4$ 4

This is the mechanism by which the optimal learning rate $d_{\text{model}}=256,\ h=4,\ L=4$ 5 is intended to remain approximately width-invariant (Khwaja et al., 19 May 2026).

For distributed training, the paper introduces dd_unit_scaling, adapting u- $d_{\text{model}}=256,\ h=4,\ L=4$ 6P to torch.compile, FSDP2/DTensor sharding, tensor parallelism, and distributed batch scaling. Unit scaling is disabled inside attention and MLP activations to preserve KV-cache behavior, and residual scaling is introduced with

$d_{\text{model}}=256,\ h=4,\ L=4$ 7

(Khwaja et al., 19 May 2026).

5. Data regime and contamination resistance

A defining property of Toto 2.0 is that its base models are pretrained on no public time-series data. The training corpus consists only of Datadog internal observability metrics and synthetic time series. The total volume is 3.40T points for the 4M and 22M models and 5.04T points for the 313M, 1B, and 2.5B models. The best base-model mixture is 42.5% observability data and 57.5% synthetic data (Khwaja et al., 19 May 2026).

Within the observability portion, the optimized frequency split is 20% Datadog 10-second metrics, 7.5% Datadog 60-second metrics, and 15% Datadog 5+ minute metrics. The real series comprise infrastructure and observability metrics such as CPU utilization, memory usage, request latency, and error rates. No customer data is used. The synthetic portion is generated from the TempoPFN prior rather than from generic stochastic processes, with the explicit goal of representing nonstationarity, abrupt changepoints, trends, and long-range dependencies (Khwaja et al., 19 May 2026).

This pretraining regime is part of a broader contamination argument. Because no public data is used in base-model pretraining, strong performance on public benchmarks is presented as cross-domain generalization rather than benchmark overlap. This is especially important for TIME, which was designed to mitigate contamination by excluding widely circulated datasets. A plausible implication is that Toto 2.0’s benchmark results are intended to function not only as performance claims but also as a methodological statement about evaluation hygiene (Khwaja et al., 19 May 2026).

6. Benchmarks, scaling evidence, release, and limitations

The empirical case for Toto 2.0 is organized around BOOM, GIFT-Eval, and TIME. On BOOM and GIFT-Eval, every model size improves on the one below it. TIME shows one minor inversion in which the 313M model slightly edges the 1B on some MASE and rank metrics, but the top three spots are still held by Toto 2.0 models (Khwaja et al., 19 May 2026).

Benchmark	Best Toto 2.0 result	Reported significance
BOOM	2.5B CRPS rank 3.88	All five sizes outrank every other foundation model on every BOOM metric
GIFT-Eval	2.5B CRPS rank 20.3	Top zero-shot foundation-model result; scaling is monotonic across sizes
TIME	2.5B CRPS rank 3.43, MASE rank 3.54, CRPS 0.532	Top three positions on every TIME metric belong to Toto 2.0

The BOOM results are especially notable because the 22M model already beats Toto 1.0, while the 4M model is competitive with substantially larger earlier systems. On GIFT-Eval, the 2.5B model is ahead of external foundation models such as PatchTST-FM r1 and Chronos-2 by the reported rank metric. On TIME, Toto 2.0’s strong results matter because TIME is explicitly framed as contamination-resistant (Khwaja et al., 19 May 2026).

The paper also reports secondary results beyond the base zero-shot story. A finetuned 2.5B-FT checkpoint is trained for 10,000 steps from a fully decayed base model, and the FnF ensemble combines all five Toto 2.0 sizes with Chronos-2, TimesFM 2.5, TiRex, FlowState, and PatchTST-FM r1 through FFORMA-style XGBoost averaging. The OWA-like training score is

$d_{\text{model}}=256,\ h=4,\ L=4$ 8

and Toto 2.0 models receive 39% of the meta-learner’s average assigned weight, ahead of Chronos-2 at 32% (Khwaja et al., 19 May 2026).

The work is also explicit about its limits. It does not present a fitted closed-form scaling law; the “scaling era” claim rests on consistent benchmark curves rather than a loss-versus-compute equation. Long-horizon extrapolation remains imperfect: on simple synthetic periodic signals, even the 2.5B checkpoint loses structure at 8192 steps, whereas classical seasonal models can extrapolate such dynamics cleanly. The data-mixture recipe is empirical rather than theory-driven, and the models still treat observability metrics as generic time series, omitting modality-specific structure such as histogram semantics, heterogeneous multivariate sampling, and very high-dimensional context selection. The paper therefore frames Toto 2.0 as a strong forecasting foundation model family, but also as an intermediate stage toward broader observability world models (Khwaja et al., 19 May 2026).