Tiny-TSM: Compact Time Series Forecasting

• Tiny-TSM is a compact time series foundation model that integrates a 23M-parameter patched encoder Transformer with causal normalization and synthetic data augmentation.
• It leverages dense next-token training with DART-Norm and coarse-grid loss, achieving state-of-the-art medium- and long-horizon forecasting on benchmark datasets.
• The model differentiates itself from similar compact TS models by balancing efficiency and performance while employing non-autoregressive inference methods like Stride-Interleaved Forecast Inference.

Tiny-TSM is a time series foundation model introduced as a small-scale, economically trained forecaster that combines a 23M-parameter patched encoder Transformer with a synthetic data generation and augmentation pipeline, SynthTS, and a causal input normalization scheme, DART-Norm. It is trained on a single A100 GPU in less than a week and is reported to achieve state-of-the-art performance on a wide range of time series benchmark datasets, particularly the best relative MSE on medium- and long-term forecasting tasks in GIFT-Eval-NF, while maintaining competitive short-term accuracy (Birkel, 24 Nov 2025).

1. Scope, identity, and nomenclature

Tiny-TSM denotes the model introduced in the 2025 paper of the same name, where “TSM” refers to a time series foundation model rather than to earlier uses of the acronym in video understanding or diffusion-model fine-tuning (Birkel, 24 Nov 2025). The model is explicitly characterized by small scale, economical training, and state-of-the-art performance, and it is presented without neural architecture search, hyperparameter tuning, or scaling up model size.

The name is potentially ambiguous in the broader literature. In video understanding, TSM denotes the Temporal Shift Module, a zero-parameter temporal operator for 2D CNN backbones (Lin et al., 2018). In diffusion modeling, TSM denotes TimeStep Master, a two-stage LoRA-based fine-tuning paradigm over timestep intervals (Zhuang et al., 10 Mar 2025). Within time-series forecasting, closely related compact models include Tiny Time Mixers (TTMs), which are TSMixer-based forecasting models with approximately 0.8M total parameters (Ekambaram et al., 2024). Tiny-TSM is distinct from all three: it is a patched encoder Transformer for forecasting, not a shift operator, not a timestep-LoRA mixture, and not a TSMixer derivative.

This distinction is consequential because Tiny-TSM’s contributions are organized around three specific mechanisms: dense next-token training enabled by strictly causal normalization, synthetic pretraining data generation, and inference-time procedures such as Stride-Interleaved Forecast Inference.

2. Architecture

Tiny-TSM uses a patched encoder-based Transformer, following the Toto architecture pattern but adapted to a smaller 23M-parameter design (Birkel, 24 Nov 2025). The input is tokenized by overlapping fixed-length patches with patch length 32. For a univariate series, the paper gives the exact mapping

$$(x_t)_{t = 0, ..., T} \in \mathbb{R} \mapsto (\bar{x}_{i,j}) \quad \text{where} \quad \bar{x}_{i,j} = x_{i*32 + j} \quad \text{for} \quad j = 1,\dots,32.$$

For multivariate series, each channel is patched independently. The input to the patch embedder is not merely the raw value; it is a concatenation of three DART-Norm features per time step, so the patch embedding linearly maps $3 \times \text{patch\_length}$ to the hidden size. The model also includes learnable embeddings for missing values, with shape patch length $\times$ hidden size, and prepends a small number of learnable pad tokens to improve behavior on short sequences. The initial input RMSNorm is removed in order to retain informative patch-level scale differences.

The encoder stack uses post-norm Transformer layers with interleaved temporal and spatial attention, in a $2\!:\!1$ ratio of time-mixing, which is causal, to space-mixing, which is non-causal. The layer structure is stated as

$$RMSNorm \mapsto MHA \mapsto Residual \mapsto RMSNorm \mapsto FFN \mapsto Residual.$$

The feed-forward network uses SwiGLU, written in the paper as

$\begin{split} & SwiGLU(x) = Swish_{W_3,\beta}(W_1 x + b) \otimes (W_2 x + c) \ & Swish_{W,\beta}(x) = W(x\sigma(\beta x)). \end{split}$

Forecasting is produced by two lightweight patch-level heads whose outputs are summed. The first is a linear head that maps the patch representation directly to the forecast horizon. The second is a cross-attention head that attends to known future covariates such as calendar features. Prediction is multivariate: all input variables are forecast jointly at each step, while known future covariates are not forecast.

A central architectural implication is that Tiny-TSM is non-autoregressive at the forecasting stage. The encoder produces patch-level forecasts directly over the desired horizon, rather than recursively generating one token at a time.

3. DART-Norm and dense next-token training

DART-Norm is the causal input normalization scheme that enables Tiny-TSM to train with dense next-token prediction without test leakage (Birkel, 24 Nov 2025). It consists of rolling standardization, explicit drift features, and target normalization anchored at the context endpoint.

The paper defines rolling statistics and normalized inputs as

$m_t = \mathrm{mean}(y_{0:t}),$

$s_t = \mathrm{std}(y_{0:t}),$

$x_t = \frac{y_t - m_t}{s_t}.$

It then augments each time step with drift features,

$d_t = \frac{m_t - m_{t-1}}{s_{t-1}},$

$3 \times \text{patch\_length}$0

The model input at time $3 \times \text{patch\_length}$1 is therefore the triplet $3 \times \text{patch\_length}$2. Future targets are normalized with the most recent causal statistics from the context boundary:

$3 \times \text{patch\_length}$3

The paper’s argument is that global z-score normalization can leak information in dense training, while rolling normalization alone can destroy identifiability by suppressing trend information. DART-Norm addresses this by exposing normalization drift explicitly. Given $3 \times \text{patch\_length}$4, the paper states that one can reconstruct $3 \times \text{patch\_length}$5 up to a global affine transform, which allows the model to emulate alternative normalization behaviors internally if needed.

This mechanism is tied directly to optimization. Dense next-token prediction increases gradient-signal density, and the paper reports roughly $3 \times \text{patch\_length}$6 faster convergence compared to masked “test-at-end” training. The training objective is point forecasting with a Huber loss; the explicit Huber formula is not provided. In addition, Tiny-TSM uses a coarse-grid loss on a subset of batches, where loss is computed only on every $3 \times \text{patch\_length}$7-th step for $3 \times \text{patch\_length}$8. This is intended to balance gradients across slow and fast dynamics and to improve medium- and long-term forecasting.

4. SynthTS and the training pipeline

The training pipeline couples DART-Norm with SynthTS, a synthetic time series generation and augmentation pipeline designed to create realistic sequences and to augment real data seamlessly (Birkel, 24 Nov 2025). SynthTS begins by sampling a base frequency and start time to construct a time index, then samples compatible frequencies that are integer multiples, with higher weights for natural scales such as daily and weekly cycles under an hourly base. The time index may also be omitted to simulate pure sequences. Batch-level parameters such as sequence length and noise level are sampled at this stage.

Base generators comprise approximately 50 families, including periodic, polynomial, and noise-based series. The examples listed in the paper include sinusoidal generators with random phase and amplitude, random polynomial generators, Gaussian and uniform noise processes, random walks, auto-regressive noisy processes, explosive processes with intermittent episodes, linear and non-linear trends, periodic processes with phase shifts, non-negative periodic series with floor at zero, integer-count series, and approximately periodic integer series. The pipeline also includes modular circular re-indexing,

$3 \times \text{patch\_length}$9

which defines a derived series $\times$0.

Augmentation is then performed in 2–5 rounds. Each round contains three steps. First, univariate expansions generate $\times$1 transforms along time, including shifts, kernel smoothing, and autoregressive transforms. Second, sparse linear combinations are applied along the feature dimension, optionally with added noise; sparsity is emphasized in order to preserve diversity and avoid pattern blurring. Third, pointwise or support-based post-transforms are applied, including

$\times$2

and amplitude modulation,

$\times$3

Missing value injection, outliers, and random spikes are also included. Outputs of the post-transform stage are appended to the pool, while intermediates from the earlier stages are discarded, and subsampling is used each round to avoid exponential growth.

The reported training configuration is intentionally economical. All experiments are conducted on a single A100 GPU, with training completed in less than a week. The optimizer is AdamW with learning rate $\times$4. The maximum input context length is 4096, the patch length is 32, the batch size is 4, and the maximum forecast horizon is 960, with per-batch horizon sampling uniform over $\times$5.

5. Evaluation and benchmark results

Tiny-TSM is evaluated on non-financial datasets from GIFT-Eval-NF, with performance reported as Mean Relative MSE and Mean Relative MAE across datasets, where “relative” denotes ratio to a seasonal naive baseline (Birkel, 24 Nov 2025). The evaluated baselines include Chronos (Base and Large), Chronos-2, Moirai-2, TiRex, TimesFM-2, TabPFN-TS, PatchTST, and N-Beats.

Model	Relative MSE (Overall / Short / Medium / Long)	Relative MAE (Overall / Short / Medium / Long)
Tiny-TSM	0.541 / 0.512 / 0.545 / 0.613	0.731 / 0.691 / 0.756 / 0.802
Chronos-2	0.542 / 0.499 / 0.566 / 0.626	0.697 / 0.656 / 0.728 / 0.767
Moirai-2	0.588 / 0.527 / 0.637 / 0.688	0.724 / 0.673 / 0.770 / 0.803
TiRex	0.594 / 0.531 / 0.637 / 0.697	0.723 / 0.674 / 0.764 / 0.802
TimesFM-2	0.647 / 0.580 / 0.681 / 0.780	0.780 / 0.724 / 0.819 / 0.899
TabPFN-TS	0.655 / 0.553 / 0.721 / 0.843	0.767 / 0.703 / 0.813 / 0.879

The paper’s main empirical claim is specific: Tiny-TSM is best on medium and long horizons under MSE among the evaluated models, while short-term MSE remains competitive. Chronos-2 is slightly better on short-term MSE, with 0.499 versus Tiny-TSM’s 0.512. Under MAE, Tiny-TSM is not best overall: Chronos-2 reports 0.697 overall MAE, compared with Tiny-TSM’s 0.731.

The paper attributes the MSE gains primarily to the combination of coarse-grid loss, DART-Norm-enabled dense training, and Stride-Interleaved Forecast Inference. It also emphasizes that these results are obtained without neural architecture search or extensive hyperparameter tuning and under a much smaller training budget than that of industrial-scale foundation models.

6. Inference, forecasting strategy, and deployment characteristics

Tiny-TSM performs non-autoregressive patch-level forecasting at inference time (Birkel, 24 Nov 2025). Several optional inference procedures are described.

The first is a symmetry-based ensemble:

$\times$6

The paper also notes that a small amount of Gaussian noise, for example with $\times$7 of input standard deviation, may be added for variance reduction. A second procedure is multivariate feature augmentation, in which transformed versions of the target, such as signed-square, signed-square-root, and kernel-smoothed variants, are appended as additional channels. This is described as often helpful for originally univariate series.

The third procedure is Stride-Interleaved Forecast Inference (SIFI), which exploits longer histories by downsampling the context into multiple strided views, running the model independently on each view, and interleaving the resulting predictions back to the original time grid. The paper gives the strided views as

$\times$8

and the interleaving rule as

$\times$9

SIFI is used only at inference. The paper suggests modest strides, such as $2\!:\!1$0, as a low-cost accuracy enhancement. This design indicates that Tiny-TSM’s forecasting quality is not solely a property of the encoder architecture; it also depends on a test-time procedure that trades additional evaluation passes for improved use of long contexts.

Deployment claims are qualitative rather than benchmarked with latency tables. The model is described as having very fast runtime due to its small size and patching, but exact inference throughput and memory footprint are not reported. The patched encoder, 23M parameter scale, and single-A100 training regime nevertheless position Tiny-TSM as substantially lighter than large industrial TSFMs.

7. Relation to adjacent compact TS models and current limitations

Tiny-TSM sits within a broader movement toward compact time-series foundation models, but it occupies a distinct design point (Birkel, 24 Nov 2025). TTMs are forecasting models with approximately 0.8M total parameters and a frozen tiny TSMixer-style backbone plus lightweight decoder/head fine-tuning, emphasizing adaptive patching, diverse resolution sampling, and resolution prefix tuning (Ekambaram et al., 2024). TSPulse is an ultra-compact 1.06M-parameter model aimed not at forecasting alone but at classification, anomaly detection, imputation, and retrieval, using dual-space masked reconstruction and dual-embedding disentanglement (Ekambaram et al., 19 May 2025). Tiny-TSM differs from both by using a patched encoder Transformer, a dense next-token forecasting objective, DART-Norm, and a synthetic data pipeline rather than a TSMixer backbone or masked-pretraining dual-space design.

This suggests a useful conceptual placement. TTMs and TSPulse prioritize extreme compactness, at or near the 1M-parameter regime, whereas Tiny-TSM targets a still-lightweight but materially larger 23M-parameter regime in exchange for stronger medium- and long-horizon forecasting under MSE. The distinction is architectural as well as operational: Tiny-TSM uses synthetic data generation and inference-time augmentation, rather than relying primarily on public-corpus pretraining or dual-space reconstruction.

Several limitations are explicit. The paper reports only one model size and does not include neural architecture search or extensive hyperparameter tuning. SIFI is only applied at inference, and the authors note that integrating coarse-to-dense strategies during training may improve long-context utilization. The model performs point forecasting only, with Huber loss; uncertainty estimation is not included. Robustness to distribution shift, missing values, and extreme events is discussed only indirectly through the inclusion of missing-value embeddings and synthetic augmentations, and formal robustness tests are not reported. Broader benchmarks beyond GIFT-Eval-NF are not evaluated. Code repository URL, license, pre-trained checkpoints, and SynthTS scripts are not provided in the paper, and seeds, exact runtime hours, and full configurations are also not reported.

Within that scope, Tiny-TSM’s central contribution is methodological rather than nominal: it demonstrates that a patched encoder Transformer with causal normalization, dense next-token training, and synthetic data can reach state-of-the-art medium- and long-horizon relative MSE on GIFT-Eval-NF under a single-A100 training budget (Birkel, 24 Nov 2025).