Fourier-Enhanced RNN: A Frequency-Aware Model

• Fourier-Enhanced RNNs are sequential models that integrate Fourier-domain features to capture periodicity, improve long-term memory, and enhance signal denoising.
• The architecture augments recurrent cells with spectral convolutions, seasonal embeddings, and compressed frequency representations for efficient multi-scale learning.
• Empirical studies show these models achieve lower forecasting errors and improved robustness compared to traditional RNNs across diverse domains.

A Fourier-Enhanced Recurrent Neural Network (RNN) is a class of sequential model that integrates explicit Fourier-domain representations or operations within recurrent architectures. These models leverage the expressive, multi-scale, and periodic structure of Fourier embeddings or spectral convolutions to address limitations of classical RNNs, such as insufficient seasonality modeling, poor long-term memory, and suboptimal signal denoising. Across power systems forecasting, physics time series modeling, signal compression, and long-horizon sequence tasks, Fourier-enhanced RNNs demonstrate improved statistical efficiency, accuracy, and gradient stability compared to traditional recurrent or operator-based alternatives.

1. Core Architectural Principles

Fourier-enhanced RNNs augment the standard recurrent backbone with one or more forms of frequency-domain computation:

• Explicit seasonal embeddings: Input features are expanded via sinusoids (Fourier harmonics) parameterized by task-relevant periods (e.g., daily, weekly, yearly), as in $X_{f,t}$ for electrical load downscaling (Chen et al., 27 Nov 2025).
• Spectral convolution: Weight matrices or cell updates are replaced/augmented with learnable convolutions in the Fourier (or related) domain, e.g., via FFT/iFFT modules for both input and hidden-state transitions (Gopakumar et al., 2023).
• Latent fusion: Fourier-encoded components are combined (typically via addition) with standard RNN latent states to yield a feature space that separates trend and seasonality.
• Self-attention: Periodic attention across latent sequences or Fourier-projected tokens further enhances intra-period dependency modeling (Chen et al., 27 Nov 2025).
• Compressed frequency representations: Low-pass filtering or indirect Fourier-space parameterization (e.g., DCT-encoded weights) reduces model dimensionality without significant loss of information (Wolter et al., 2018, Basterrech et al., 2022).

These design elements reduce the burden on recurrence mechanisms to infer periodic or smooth structure, enabling both improved learning of long-term dependencies and more interpretable decomposition of latent features.

2. Mathematical Formulations and Variants

2.1 Explicit Fourier Feature Injection

In power load downscaling, the input per period $t$ includes both an aggregate scalar $x_{0,t} \in \mathbb{R}$ and a Fourier feature tensor $X_{f,t} \in \mathbb{R}^{K \times 2F}$, with $K$ sub-periods and $F$ harmonics: $$X_{f,t}[s,2k-2] = \sin(\omega_k \cdot (t + s/K)), \qquad X_{f,t}[s,2k-1] = \cos(\omega_k \cdot (t + s/K)),$$ where $\omega_k = \frac{2\pi k}{P}$ for a base period $P$ (e.g., 7 days, 365 days) (Chen et al., 27 Nov 2025). This is projected via a trainable linear operator and fused additively with RNN-latent activations prior to output or attention.

2.2 Spectral RNN Cells

An alternative approach is to replace RNN cell operations with spectral filtering: $$\hat{x}_t = \mathcal{F}(x_t), \quad \hat{h}_{t-1} = \mathcal{F}(h_{t-1})$$

$t$0

$t$1

where $t$2 and $t$3 denote forward and inverse (truncated) FFT, $t$4, $t$5 are spectral convolutions, and $t$6, $t$7 are real-space linear maps (Gopakumar et al., 2023).

2.3 Indirect and Compressed Frequency Parameterization

For model size and optimization, reservoir weights or inputs/outputs can be compressed via discrete cosine transforms (DCT) or frequency truncation, e.g.:

• DCT compression of reservoir weights in Echo State Networks, optimizing only a few low-frequency coefficients for thousands of connections (Basterrech et al., 2022).
• STFT low-pass representation of input windows, reducing RNN steps and weight matrices while suppressing high-frequency noise (Wolter et al., 2018).

2.4 Frequency-Augmented Memory

Some variants, such as the Fourier Recurrent Unit (FRU), use dynamic Fourier-summation memory states: $t$8 with $t$9 a diagonal block matrix comprised of cosines for each frequency and channel, so that $x_{0,t} \in \mathbb{R}$0 encodes a running sum of $x_{0,t} \in \mathbb{R}$1 modulated by Fourier bases (Zhang et al., 2018).

3. Empirical Performance and Use Cases

Fourier-enhanced RNNs have demonstrated strong empirical gains across several domains:

Task/Setting	Model	Key Findings	Reference
PJM load downscaling	Fourier-RNN	20% lower mean RMSE than Prophet with seasonality+LAA; flatter errors	(Chen et al., 27 Nov 2025)
Physics PDE surrogate (Navier–Stokes, Wave Eqn)	Spectral Fourier-RNN	Order-of-magnitude lower MSE vs RNNs/FNO at high noise levels	(Gopakumar et al., 2023)
Chaotic Mackey-Glass, Lorenz, sunspot forecasting	DCT-compressed EvoESN	NRMSE improved by multiple orders over classical ESN, with 1000x smaller parameter space	(Basterrech et al., 2022)
Power load, motion prediction	STFT-RNN	MSE reduction and 5x–10x speedup over time-GRU; parameter sparsity	(Wolter et al., 2018)
Long-sequence MNIST, IMDB, etc.	FRU, Oscillatory F-NN	Faster convergence, constant gradients, parameter reduction	(Zhang et al., 2018); (Han et al., 2021)

These improvements are attributed to more effective representation of long-period seasonality, reduction in parameter space via compression, improved robustness to noise, and better calibration of uncertainty intervals.

4. Theoretical Properties and Expressivity

Analytic studies of Fourier-enhanced RNNs establish several salient properties:

• Gradient stability: In the FRU, gradients through time are bounded independently of sequence length, eliminating both vanishing and exploding gradient phenomena in the linearized regime (Zhang et al., 2018). This is in contrast to standard RNNs, where gradient norms scale exponentially with $x_{0,t} \in \mathbb{R}$2, and to Statistical Recurrent Units (SRU), which only partially mitigate this via multi-scale exponential memory.
• Expressivity of sparse Fourier basis: Sparse trigonometric expansions (i.e., sums of cosines with few frequencies) can approximate arbitrary-degree polynomials on finite intervals to arbitrary precision, a property not shared by exponential decays or moving averages used in standard RNNs/SRUs (Zhang et al., 2018).
• Decomposition of trend and seasonality: Linear additive fusion cleanly decouples trend learning (RNN) from periodicity (Fourier), providing a mechanism for interpretable modeling (Chen et al., 27 Nov 2025).

A plausible implication is that Fourier-based memory or parametrization enables natural multi-scale learning without increasing parameter count or depth, crucial for long-horizon tasks and data with prominent periodic structure.

5. Training Methodologies and Optimization Strategies

Fourier-enhanced RNNs employ standard loss objectives (e.g., mean-squared error), but also often include regularization tailored to the frequency domain, such as harmonic penalization to prevent overfitting to high-frequency components (Chen et al., 27 Nov 2025), or parameter norm constraints to induce spectral smoothness (Wolter et al., 2018).

Optimization leverages Adam or similar methods, often with gradient clipping and early stopping. In spectral-parameterized architectures, all parts of the computation graph—including FFT/iFFT, STFT, DCT, and windowing functions—are made differentiable to support end-to-end training (Wolter et al., 2018); in echo-state networks, only the (compressed) frequency-domain weights are evolved, typically via genetic algorithms in reduced space (Basterrech et al., 2022).

Notably, oscillatory neuron models replace the need for backpropagation through time entirely, as the update per timestep is locally differentiable and fully parallelizable (Han et al., 2021).

6. Limitations, Challenges, and Extensions

Identified limitations include:

• Dependence on Gaussian residuals: In load forecasting applications, confidence intervals may under- or over-cover if residuals are non-Gaussian (Chen et al., 27 Nov 2025).
• Loss of high-frequency content: Frequency truncation, necessary for computational efficiency and denoising, can omit significant high-frequency structure when present in the data (Gopakumar et al., 2023).
• Applicability to non-uniform or non-Euclidean domains: Purely spectral methods are less well-suited to tasks involving irregular time grids or spatially non-Euclidean data; extensions to wavelets or graph-Fourier bases are possible (Gopakumar et al., 2023).
• Model size vs. hardware fit: Some approaches (e.g., O-FNN) sacrifice general flexibility for extreme parameter efficiency and parallelizability, best suited only for many-to-one tasks or dedicated edge hardware (Han et al., 2021).

Key extensions and research directions include:

• Full probabilistic output layers for uncertainty quantification (Chen et al., 27 Nov 2025).
• Spatio-temporal attention and multi-site downscaling (Chen et al., 27 Nov 2025).
• Adaptive, data-dependent spectral filtering, or learning basis functions.
• Incorporation of Fourier-domain techniques in reservoir computing or hybrid operator architectures (Gopakumar et al., 2023, Basterrech et al., 2022).

7. Relationship to Broader Time Series and Operator Learning Paradigms

Fourier-enhanced RNNs form part of a larger trend in sequence modeling, leveraging operator-theoretic and frequency-domain priors to overcome fundamental shortcomings of classic recurrence-based models. Unlike vanilla RNNs, which are limited by memory capacity and gradient flow through deep time, or Markovian operator networks (such as FNO), which lack persistent memory, these hybrids achieve a balance of global convolutional expressivity, multi-scale periodicity, and effective long-term recursion (Gopakumar et al., 2023, Zhang et al., 2018).

This synthesis yields architectures that are more robust to noise, more parameter efficient, and capable of learning rich structure from modest data—characteristics which have motivated adoption in power systems, physics-informed learning, and compressive time series modeling.

---

In summary, Fourier-Enhanced Recurrent Neural Networks explicitly incorporate frequency-domain knowledge to improve trend and seasonality modeling, robustness, and training dynamics in sequential modeling tasks. They have been empirically validated in real-world and synthetic benchmarks across multiple domains, and their core principles continue to inform the design of next-generation models for long-horizon and multi-scale time series analysis (Chen et al., 27 Nov 2025, Gopakumar et al., 2023, Wolter et al., 2018, Basterrech et al., 2022, Han et al., 2021, Zhang et al., 2018).