FreezeTST: Frozen Time-Series Forecasting

• The paper introduces FreezeTST, a long-sequence forecasting model that interleaves trainable Transformer layers with frozen random-feature modules.
• It achieves parameter efficiency by freezing nearly half of the encoder layers, lowering backward computation without sacrificing performance.
• Empirical results on multiple datasets demonstrate that FreezeTST rivals PatchTST, offering robust long-term predictions and reduced training overhead.

Searching arXiv for FreezeTST and closely related time-series Transformer baselines. FreezeTST is a long-term time-series forecasting model introduced in “Frozen in Time: Parameter-Efficient Time Series Transformers via Reservoir-Induced Feature Expansion and Fixed Random Dynamics” (Singh et al., 25 Aug 2025). It combines a patch-based Transformer encoder with frozen random-feature or reservoir-like blocks in order to improve efficiency and robustness on long horizons, a regime in which standard Transformers can be expensive and statistically brittle. In the formulation used by the paper, the task is to map a multivariate history $\mathbf{x}_{1:T}\in\mathbb{R}^{T\times d}$ to an $H$-step forecast, with the model written as $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$ and trained by direct multi-step MSE (Singh et al., 25 Aug 2025). The central architectural idea is to interleave standard trainable Transformer layers with fixed random dynamics: either an explicit Echo-State Component (ESC) or, in the main practical method, Transformer blocks frozen at random initialization and treated as fixed nonlinear random feature maps. The paper positions this design against specialized long-sequence forecasters such as Informer, Autoformer, and PatchTST, arguing that efficiency can be obtained through reduced optimization cost rather than through a new attention kernel (Singh et al., 25 Aug 2025).

1. Definition and problem setting

FreezeTST is defined for long-sequence time-series forecasting (LSTF), where the model receives an input window of length $T$ and predicts the next $H$ steps of a multivariate series (Singh et al., 25 Aug 2025). The paper emphasizes two familiar difficulties for vanilla Transformers in this regime. First, full self-attention has $O(T^2)$ time and memory complexity as input length grows. Second, self-attention has only weak built-in temporal bias, because positional encodings annotate order but attention itself is content-based; this can be brittle when chronology, seasonality, and long memory are central (Singh et al., 25 Aug 2025).

The model is proposed as a response to the gap between standard Transformer expressivity and the practical demands of long-horizon forecasting. FreezeTST incorporates reservoir computing principles without training a recurrent reservoir. Its terminology is explicit. “Parameter-efficient” means that a substantial fraction of encoder layers are frozen, reducing the number of trainable parameters and the backward-pass cost, often by about half. “Reservoir-induced feature expansion” means that fixed random maps expand the hidden representation into a richer nonlinear basis without learned weights in those layers. “Fixed random dynamics” means weights drawn once at initialization and never updated thereafter (Singh et al., 25 Aug 2025).

This design places FreezeTST closest to the PatchTST family conceptually, because it uses patch tokenization and channel-independent processing, but differs by turning selected encoder blocks into fixed random-feature modules. The paper’s broader thesis is that frozen random transforms provide long-memory and nonlinear basis functions “for free,” while trainable attention layers learn how to query and combine those features adaptively (Singh et al., 25 Aug 2025).

2. Encoder architecture and forecasting pipeline

The end-to-end pipeline follows PatchTST-style tokenization (Singh et al., 25 Aug 2025). Each channel is processed independently as a univariate sequence and split into patches. For channel $k\in\{1,\dots,d\}$ with $\mathbf{x}^{(k)}\in\mathbb{R}^T$, the $i$-th patch is

$$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$

for $H$0, with

$H$1

The reported experiments use patch length $H$2 and stride $H$3 (Singh et al., 25 Aug 2025). This reduces the effective sequence length from raw time steps to patch tokens and thereby reduces attention cost.

Each patch is linearly embedded into model dimension: $H$4 with $H$5 and $H$6, giving a per-channel patch sequence $H$7 (Singh et al., 25 Aug 2025). These embeddings are then enriched with deterministic positional encodings before entering the encoder, although the exact positional encoding formula is not specified in the provided material.

The encoder is a stack of Transformer blocks. For one channel,

$H$8

A block is described as MHSA plus FFN with residual connections and layer normalization; a simplified expression used in the paper is

$H$9

Channels are processed independently, exactly in the PatchTST spirit, so attention is along the patch dimension only. The paper therefore states that cost is reduced from roughly $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$0 to $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$1 per channel stream (Singh et al., 25 Aug 2025).

The prediction head maps each channel’s hidden representation to an $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$2-step forecast, using the last hidden state for each channel and then concatenating outputs across channels to form $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$3 (Singh et al., 25 Aug 2025). The paper does not provide a more explicit formula for the head in the provided text.

3. Reservoir-induced feature expansion and partial freezing

FreezeTST introduces the “frozen reservoir” in two related forms (Singh et al., 25 Aug 2025). The first is an explicit Echo-State Component, presented as an explanatory bridge to reservoir computing. The ESC maintains a hidden state $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$4 updated by

$$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$5

where $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$6 is the leak rate and the recurrent weights are scaled so that $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$7, ensuring contractive dynamics and fading memory (Singh et al., 25 Aug 2025). The reservoir state is projected back to model dimension by a learned linear readout. This literal ESC is reported to help empirically, but because it is sequentially unrolled it hurts parallelism and can increase wall-clock time.

The second mechanism is the final FreezeTST method: selected Transformer encoder blocks are frozen immediately after random initialization and thereafter interpreted as fixed nonlinear, variance-preserving random maps (Singh et al., 25 Aug 2025). Let $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$8 denote frozen block indices and $$\hat{\mathbf{x}}_{T+1:T+H}=\mathcal{F}_\theta(\mathbf{x}_{1:T})$$9 trainable ones. Frozen blocks are not recurrent ESNs in the literal sense; rather, they are treated as reservoir-like random feature generators embedded within the encoder. The trainable neighboring blocks act as adaptive readouts over these random features.

The canonical configuration is a patchwise Transformer backbone with every second layer frozen in general. In the main experiments, the principal instantiation is a 3-layer encoder with the middle layer frozen, yielding the alternating pattern

$T$0

This interleaving is architecturally central. The frozen blocks are not merely a front-end preprocessor; they modify hidden states inside the encoder between trainable attention layers (Singh et al., 25 Aug 2025).

The trainable and frozen components are sharply separated. Trainable parameters include the patch embedding, the trainable encoder blocks, the output prediction head, and likely normalization affine parameters within trainable blocks. Frozen parameters are the selected encoder blocks, fixed at Xavier initialization. In the ESC variant, $T$1 and $T$2 are fixed random weights, while the readout back to model dimension is learned (Singh et al., 25 Aug 2025).

4. Theoretical formulation and stability claims

The paper gives a theoretical interpretation of frozen Transformer blocks as fixed random-feature expansions supported by variance-preserving initialization and Lipschitz control (Singh et al., 25 Aug 2025). Attention and FFN weight matrices are initialized with zero-mean i.i.d. variance

$T$3

and with Xavier or Glorot initialization

$T$4

For fixed $T$5,

$T$6

This is used to argue variance preservation in expectation (Singh et al., 25 Aug 2025).

To obtain stability, the paper imposes a spectral norm bound: frozen blocks are rescaled once so that $T$7, and trainable blocks are kept with spectral norm at most $T$8 during training. If

$T$9

and every block is 1-Lipschitz, then

$H$0

For a differentiable loss $H$1,

$H$2

The stated interpretation is that gradient explosion or collapse is prevented regardless of how many blocks are frozen or where they are placed (Singh et al., 25 Aug 2025).

For the explicit ESC, the paper derives an exponential forgetting result. Defining

$H$3

if two input sequences differ only at time $H$4, then

$H$5

This yields an effective receptive field

$H$6

The paper gives two concrete memory-related parameter settings in different contexts: $H$7, yielding $H$8 and $H$9, and an experimental statement that performance for the ESC-augmented variant peaks around $O(T^2)$0 with effective memory approximately $O(T^2)$1 steps (Singh et al., 25 Aug 2025). The paper itself notes that these settings are not fully reconciled.

The theoretical discussion also links the frozen-block view to random-feature theory and reservoir universality, arguing that random expansions followed by learned contractions can approximate rich function classes with error decaying like $O(T^2)$2 in width $O(T^2)$3, though the exact theorem statement is cited from prior work rather than derived in the paper (Singh et al., 25 Aug 2025). This suggests that FreezeTST should be understood less as a literal ESN-Transformer hybrid in its final form than as a partially frozen patchwise Transformer motivated by reservoir computing.

5. Computational profile and empirical results

FreezeTST’s efficiency claim is specific: inference complexity is unchanged relative to a same-depth unfrozen backbone, because frozen layers are still evaluated in the forward pass (Singh et al., 25 Aug 2025). The computational savings arise during training through the absence of gradient updates for frozen layers, a reduced backward graph, and fewer trainable parameters. If a vanilla $O(T^2)$4-layer backbone has $O(T^2)$5 parameters per block, then under alternate freezing the trainable budget is

$O(T^2)$6

which is roughly half the encoder trainable budget (Singh et al., 25 Aug 2025).

The evaluation covers seven public LSTF datasets: ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and ILI (Singh et al., 25 Aug 2025). Reported dataset statistics are ETTh1: 17,420 time steps, 7 features, hourly; ETTh2: 17,420, 7, hourly; ETTm1: 69,680, 7, 15-min; ETTm2: 69,680, 7, 15-min; Weather: 52,696, 21, 10-min; Electricity: 26,304, 321, hourly; and ILI: 966, 7, weekly. Forecast horizons are $O(T^2)$7 for ETT, Weather, and Electricity, and $O(T^2)$8 for ILI. Input lengths are $O(T^2)$9 for all except ILI, where $k\in\{1,\dots,d\}$0. Metrics are MSE and MAE. Results are averaged over 3 runs for every dataset except Electricity, where only 1 run was possible due to GPU memory constraints. Implementation uses PyTorch 1.11.0 and a single NVIDIA Quadro P5000 GPU with 16 GB VRAM; hyperparameters are tuned on validation with Bayesian optimization for up to 60 trials (Singh et al., 25 Aug 2025).

Across multivariate forecasting, the paper reports that FreezeTST ranks first or second on six of seven datasets and never outside the top three (Singh et al., 25 Aug 2025). Relative to PatchTST, the results are generally extremely close. On ETTh1 at $k\in\{1,\dots,d\}$1, FreezeTST reports $k\in\{1,\dots,d\}$2 versus PatchTST at $k\in\{1,\dots,d\}$3; at $k\in\{1,\dots,d\}$4, $k\in\{1,\dots,d\}$5 versus $k\in\{1,\dots,d\}$6; and at $k\in\{1,\dots,d\}$7, $k\in\{1,\dots,d\}$8 versus $k\in\{1,\dots,d\}$9. On ETTh2 at $\mathbf{x}^{(k)}\in\mathbb{R}^T$0, both report MSE $\mathbf{x}^{(k)}\in\mathbb{R}^T$1, with MAE $\mathbf{x}^{(k)}\in\mathbb{R}^T$2 for FreezeTST versus $\mathbf{x}^{(k)}\in\mathbb{R}^T$3 for PatchTST, and at $\mathbf{x}^{(k)}\in\mathbb{R}^T$4, FreezeTST reports $\mathbf{x}^{(k)}\in\mathbb{R}^T$5 versus $\mathbf{x}^{(k)}\in\mathbb{R}^T$6. On ETTm2, which the paper presents as one of the stronger cases for FreezeTST, results include $\mathbf{x}^{(k)}\in\mathbb{R}^T$7 versus $\mathbf{x}^{(k)}\in\mathbb{R}^T$8 at $\mathbf{x}^{(k)}\in\mathbb{R}^T$9, and $i$0 versus $i$1 at $i$2 (Singh et al., 25 Aug 2025).

Weather is reported as a case where PatchTST is somewhat better numerically, though FreezeTST remains competitive; for example, at $i$3 FreezeTST reports $i$4 versus PatchTST’s $i$5, and at $i$6 $i$7 versus $i$8 (Singh et al., 25 Aug 2025). Electricity is nearly tied, including identical MSE of $i$9 at $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$0 and $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$1 versus $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$2 at $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$3 (Singh et al., 25 Aug 2025). For ILI, the paper explicitly notes that the table presentation is not fully clean and that the surrounding text is somewhat inconsistent, though FreezeTST is still described as competitive.

A paired Wilcoxon signed-rank test is reported to find no statistically significant difference between FreezeTST and PatchTST at $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$4 (Singh et al., 25 Aug 2025). This is important for interpreting the empirical claims: the paper does not claim a decisive accuracy gap over PatchTST so much as comparable or slightly better forecasting with reduced trainable compute.

6. Ablations, interpretation, and relation to adjacent methods

The ablation studies are central to the paper’s interpretation of partial freezing (Singh et al., 25 Aug 2025). In a 5-layer PatchTST backbone, the authors compare five freezing schemes: $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$5 (no freezing), $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$6 (all encoders frozen, only the head trainable), $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$7 (alternate layers frozen), $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$8 (first layer frozen), and $$\mathbf{p}_{i}^{(k)}=\bigl[x^{(k)}_{(i-1)s+1},\dots,x^{(k)}_{(i-1)s+p}\bigr]^\top\in\mathbb{R}^{p},$$9 (first and last frozen). On ETTh1 at $H$00, reported MSEs are $H$01 for $H$02, $H$03 for $H$04, $H$05 for $H$06, $H$07 for $H$08, and $H$09 for $H$10. At $H$11, the values are $H$12, $H$13, $H$14, $H$15, and $H$16, respectively (Singh et al., 25 Aug 2025). The paper characterizes this as striking because even freezing all encoder layers performs close to baseline while reducing trainable parameters by 63.6%. The authors conclude that freezing alternate layers gives the best overall performance-efficiency trade-off.

Depth ablations on ETTh1 with $H$17 show that FreezeTST performs best around 4 to 6 encoders and deteriorates thereafter, which the paper interprets as overfitting (Singh et al., 25 Aug 2025). For the ESC variant, leak $H$18 is varied from $H$19 to $H$20 and spectral radius $H$21 from $H$22 to $H$23, with best performance occurring around $H$24, where effective memory approximately matches the forecast horizon. Reservoir size ablations show that 500 units work best, while 300 and 1000 are close, with MSE differences under 0.005 (Singh et al., 25 Aug 2025). Small standard deviations in the efficiency table are presented as evidence that freezing does not introduce major instability.

The efficiency tables reinforce the parameter-efficiency claim. On ETTh1, a vanilla 3-layer PatchTST reports 0.8M trainable parameters, 13.00 min total train time, and 7.80 s/epoch, whereas FreezeTST with 3 layers and 1 frozen reports 0.7M trainable parameters, ratio 0.875, 11.83 min total train time, and 7.10 s/epoch. A 4-layer FreezeTST with 2 frozen layers reports 0.7M parameters, ratio 0.778, 11.40 min total train time, and 6.84 s/epoch; a 5-layer FreezeTST with 2 frozen reports 0.8M parameters, ratio 0.727, 12.40 min, and 7.44 s/epoch (Singh et al., 25 Aug 2025). Elsewhere, the paper states that encoder trainable parameters can be cut by up to 50%, wall-clock training reduced by 20–30%, and that an 8-layer FreezeTST trains $H$25 faster than an equally deep PatchTST in one GPU profiling result (Singh et al., 25 Aug 2025). These figures are reported claims from the paper rather than fully standardized complexity proofs.

Within the broader forecasting literature, FreezeTST is positioned against two lines of prior work: efficient-attention or structure-aware Transformers, such as Informer and Autoformer, and patch-based forecasting, especially PatchTST (Singh et al., 25 Aug 2025). Relative to models that redesign the attention operator, FreezeTST is novel in a different sense: it keeps a familiar patchwise Transformer and modifies the optimization architecture by freezing some layers. Relative to reservoir computing, it borrows fading-memory intuition, fixed random dynamics, and trainable readout over untrained temporal features, but retains self-attention and FFN modules around the fixed maps. Relative to random-feature models, the frozen layers resemble deep fixed nonlinear projections inserted between adaptive layers (Singh et al., 25 Aug 2025).

A useful disambiguation is that FreezeTST is unrelated to “Freeze then Train,” a robustness method for representation learning under spurious correlations and test-time probing (Ye et al., 2022). The name “FreezeTST” in the forecasting literature specifically refers to the partially frozen time-series Transformer of (Singh et al., 25 Aug 2025), not to the FTT method of (Ye et al., 2022).

7. Reproducibility, limitations, and significance

The paper includes two formal propositions and proofs: one on non-expansiveness and gradient bounds for arbitrary frozen or trainable layer placements, and one on exponential forgetting and effective receptive-field length for the explicit reservoir dynamics (Singh et al., 25 Aug 2025). It does not include a pseudocode algorithm box in the provided text. Reproducibility notes indicate that baselines were retrained under a unified protocol, hyperparameters were tuned with Bayesian optimization, and supplemental material is said to include final hyperparameters, random seeds, and data-processing code; no repository URL is given in the provided material (Singh et al., 25 Aug 2025).

Several caveats are explicitly acknowledged. The strongest empirical case is on ETT and Electricity-style benchmarks; on Weather and some ILI settings, gains over PatchTST are less clear (Singh et al., 25 Aug 2025). Reporting around ILI is noted as inconsistent, and the ESC discussion contains unreconciled parameter examples for effective memory. The paper also emphasizes that the final method’s “reservoir” interpretation is broader than a literal recurrence: FreezeTST is not always an actual ESN block, and is more often a partially frozen Transformer encoder (Singh et al., 25 Aug 2025). Because the backbone remains channel-independent in the PatchTST style, explicit cross-channel attention is not central to the method.

The significance of FreezeTST lies in its reframing of efficiency for long-horizon forecasting. Rather than reducing inference complexity via sparse or low-rank attention, it reduces training cost through partial freezing while leaving the forward pattern unchanged (Singh et al., 25 Aug 2025). This suggests a distinct design principle for LSTF systems: randomness and freezing can function as a useful prior inside the encoder, supplying stable nonlinear feature expansion that trainable attention layers then learn to query. A plausible implication is that FreezeTST’s main contribution is methodological and architectural rather than operator-theoretic. It retains a standard Transformer computation graph where needed, but repurposes part of the network as fixed internal memory-like structure, thereby connecting patch-based time-series Transformers to reservoir computing in a practically implementable form (Singh et al., 25 Aug 2025).