---
title: 'Fredformer Model: Frequency-Aware Forecasting'
url: https://www.emergentmind.com/topics/fredformer-model
type: topic
---

# Fredformer Model: Frequency-Aware Forecasting

Fredformer is a Transformer-based framework specifically designed to mitigate the frequency bias exhibited by mainstream Transformers in the context of time series forecasting. Standard Transformer models, despite their empirical success, consistently favor low-frequency, high-amplitude features while suppressing high-frequency, low-amplitude signal components. Fredformer introduces a frequency-domain processing pipeline involving normalization and localized channel-wise attention, equalizing the model's focus across frequency bands and preserving critical high-frequency information. A lightweight Nyström-approximated variant further enables scalability to highly multivariate settings while maintaining computational efficiency and competitive predictive accuracy [2406.09009].

## 1. Frequency Bias in Transformer-Based Time Series Forecasting

Extensive analyses demonstrate that classic Transformers trained on time series $X \in \mathbb{R}^L$ (and its multivariate extensions $X \in \mathbb{R}^{C \times L}$) tend to reconstruct the low-frequency structure but consistently fail to recover mid- to high-frequency details in $\hat{Y}$ with respect to $Y_{\text{true}}$. This phenomenon is formalized through synthetic experiments wherein signals are composed of $N$ "key" frequency components with ordered amplitudes $|\tilde{A}_1| \geq |\tilde{A}_2| \geq \dots$. The per-frequency relative error,
$$
\Delta_k = \frac{|A'_k - \hat{A}_k|}{|\hat{A}_k|},
$$
is shown to be inversely correlated to the amplitude proportion,
$$
P(\tilde{A}_k) = \frac{|\tilde{A}_k|}{\sum_{n=1}^N |\tilde{A}_n|};
$$
specifically, larger $P(\tilde{A}_k)$ predicts smaller $\Delta_k$, demonstrating that Transformer's loss landscape prioritizes features with greater frequency-domain energy. This outcome persists regardless of frequency ordering, verifying that the bias hinges solely on amplitude magnitudes, not spectral location.

## 2. Architectural Components of Fredformer

Fredformer inserts a structured frequency-domain pipeline into the Transformer, comprised of four principal stages:

**(i) DFT-to-IDFT Backbone:**  
The model first converts the input $X \in \mathbb{R}^{C \times L}$ into the frequency domain using the Discrete Fourier Transform (DFT),
$$
A_k = \frac{1}{L} \sum_{l=1}^{L} X_l e^{-i 2\pi k l / L}.
$$
Frequency-domain processing ensues, and the output spectrum $A' \in \mathbb{C}^{C \times L}$ is transformed back via the inverse DFT (IDFT).

**(ii) Frequency Refinement and Normalization:**  
The frequency axis of $A$ is partitioned into $N$ patches $W_1, \ldots, W_N \subset \mathbb{C}^{C \times S}$, each of width $S = L/N$. Amplitudes within each patch are normalized so that
$$
W^*_n = \sigma(W_n), \qquad \text{with} \ \max(|W^*_n|) = 1 \ \forall n,
$$
removing the amplitude disparities across bands and debiasing subsequent attention computations.

**(iii) Frequency-local Independent Modeling:**  
Each normalized patch $W^*_n$ is processed by a local Transformer encoder, treating each channel (row) as a token. Channel-wise (not temporal or cross-band) attention is applied:
$$
\text{Attention}_n = \text{Softmax} \left( \frac{W^*_n W^q_n (W^*_n W^k_n)^{T}}{\sqrt{d}} \right) W^*_n W^v_n.
$$
This isolates channel interactions at each frequency, ensuring that $\Delta_k$ is no longer determined by $P(\tilde{A}_k)$.

**(iv) Frequency-wise Summarization and IDFT:**  
Refined outputs from all patches are concatenated or projected to reconstruct the complete frequency matrix $A'$, which is then returned to the time domain via IDFT.

## 3. Lightweight Variant: Nyström-Fredformer

To address the $O(C^2)$ per-patch complexity of self-attention with large channel counts, Fredformer integrates a Nyström-based approximation. Here, $m$ landmark rows/columns are selected within each patch to define $A_S \in \mathbb{R}^{m \times m}$. The original attention matrix
$$
\text{Softmax}(QK^T)
$$
is approximated by
$$
F_S A_S^+ B_S,
$$
where $A_S^+$ is the Moore–Penrose inverse of $A_S$, and $F_S$ and $B_S$ are computed from the softmax between full and downsampled queries/keys. This innovation reduces frequency patch complexity from $O(C^2 L / P)$ to $O(CL / P)$, enabling deployment on datasets with hundreds of channels while yielding negligible accuracy loss.

## 4. Forecasting Benchmarks and Experimental Validation

Fredformer and Nyström-Fredformer are validated on eight standard multivariate benchmarks: Weather (21 channels), Electricity ($C=321$), ETTh1/ETTh2/ETTm1/ETTm2 ($C=7$), Solar-Energy ($C=137$), and Traffic ($C=862$), with lookback $L=96$ and prediction horizons $H \in \{96,192,336,720\}$. Comparisons include 11 baselines such as Autoformer, FEDformer, Pyraformer, Crossformer, PatchTST, Stationary, iTransformer, DLinear, RLinear, TiDE, and TimesNet.

Averaged across $8$ datasets and $4$ horizons ($32$ settings), Fredformer achieves the lowest mean squared error (MSE) in 60 out of 80 cases. Typical results:

| Dataset | Fredformer (MSE/MAE) | 2nd Best     | Next Best   |
|---------|----------------------|--------------|-------------|
| ETTh1   | 0.435 / 0.426        | 0.454 / 0.447 (iTrans) | 0.446 / 0.434 (PatchTST) |
| ETTh2   | 0.365 / 0.393        | 0.383 / 0.407 (PatchTST) | 0.374 / 0.398 (iTrans)   |
| ECL     | 0.175 / 0.269        | 0.178 / 0.270 (iTrans) | 0.192 / 0.296 (Autoformer) |

## 5. Ablation, Sensitivity, and Computational Analysis

Ablation experiments on ETTh1 and Weather datasets identify two critical components: (1) channel-wise attention (CW) and (2) frequency refinement/patching (FR). Removal of either results in significant degradation:

| Setting    | ETTh1 MSE / MAE | Weather MSE/MAE |
|------------|-----------------|-----------------|
| Full       | 0.384 / 0.396   | 0.246 / 0.273   |
| –No-CW     | 0.418 / 0.419   | 0.262 / 0.290   |
| –No-FR     | 0.539 / 0.485   | 0.293 / 0.322   |

Finer patch divisions correlate with better MSE (ETTh1, $S \in \{8,16,32,L\}$ yields $\text{MSE} = \{0.417, 0.425, 0.440, 0.449\}$). Computationally, Fredformer requires approximately 408MB GPU memory on ETTh1 ($C=7$) and 4.3GB on Electricity ($C=321$); Nyström-Fredformer reduces the latter to 3.3GB (a 25% reduction) while maintaining equivalent MSE (0.212 vs. 0.213).

## 6. Significance and Implications

Fredformer overcomes Transformer frequency bias by modeling directly in the frequency domain, explicitly normalizing amplitudes per band, and applying channel-wise attention. This equalizes the treatment of high-frequency, low-energy components that are overlooked by standard architectures. The lightweight Nyström-Fredformer extends the applicability to domains featuring hundreds of channels, with minimal computational or predictive penalty. A plausible implication is that frequency-domain architectural correction could be adapted to other forecasting or generative tasks exhibiting energy-driven feature suppression. Extensive multi-benchmark evaluation demonstrates robustness and consistent superiority to prior state-of-the-art paradigms across both accuracy and scalability dimensions [2406.09009].

Source: https://www.emergentmind.com/topics/fredformer-model