FFT-Based Acceleration in Sequence Models

• FFT-based acceleration reduces quadratic scaling to $\mathcal{O}(n\log n)$, boosting efficiency in deep neural models for long sequences.
• Core implementation involves replacing self-attentions with spectral filtering and FFT-based convolutions for increased speed and reduced memory usage.
• Applicable in NLP, vision, and genomics, these methods enable models to process larger context windows with substantial throughput improvements.

Fast Fourier Transform (FFT)-based acceleration in sequence models encompasses a family of approaches leveraging the $\mathcal{O}(n\log n)$ complexity of FFTs to replace or augment the quadratic scaling bottlenecks of self-attention, convolutions, and kernel operations in deep neural architectures for long sequences. These methods span from model-level algorithmic innovations to low-level hardware optimizations, yielding significant throughput, memory, and energy benefits for tasks in natural language, vision, genomics, and other domains requiring large context windows or high-resolution input.

1. Core Principles of FFT-Based Acceleration in Sequence Models

FFT-based acceleration exploits the fact that convolutions, Toeplitz matrix-vector products, and certain forms of global mixing can be computed efficiently in the frequency domain using the FFT. For sequences of length $n$, these operations, which are $\mathcal{O}(n^2)$ in the time domain, reduce to $\mathcal{O}(n\log n)$ due to the properties of the FFT and the convolution theorem:

$$\text{conv}(x, k) \xleftrightarrow{\text{FFT}} \text{IFFT}(\text{FFT}(x)\odot \text{FFT}(k))$$

The FFT-centric approach has been deployed in several classes of sequence models:

• Spectral Token Mixing: Replacing attention or mixing modules with spectral filtering defined and modulated in the frequency domain, e.g., SPECTRE/FFTNet (Fein-Ashley et al., 25 Feb 2025).
• FFT-Accelerated Convolutions: Implementing long convolutions (kernel as long as the input) in models such as Hyena using efficient, hardware-aware FFT block decompositions, e.g., FlashFFTConv (Fu et al., 2023).
• Toeplitz and Kernel Methods: Approximating or directly applying Toeplitz matrix-vector products and low-rank kernel interactions in the frequency domain (Moreno et al., 2023).
• Redundancy Removal: Leveraging the tendency of deep hidden states to concentrate in low-frequency bands and applying DCT (via FFT) for lossy or lossless compression, as in Fourier Transformer (He et al., 2023).

2. Spectral Token Mixing: Architectural Techniques and Algorithmic Formulations

One canonical architectural pattern is SPECTRE (called FFTNet) (Fein-Ashley et al., 25 Feb 2025), which replaces the dot-product self-attention block in Transformers with an efficient per-head spectral block. For an input $X \in \mathbb{R}^{n\times d}$:

1. Forward Real FFT: Compute $F = \mathrm{FFT}(X) \in \mathbb{C}^{n\times d}$, where FFT is applied along the sequence axis for each channel.
2. Content-Adaptive Spectral Gate: Use a learned, context-dependent spectral mask $W$ (obtained via MLP on global context) to adaptively modulate the spectrum: $\widetilde F = F \odot W$.
3. Complex Nonlinearity (modReLU): Apply modReLU activation for expressivity: $\widehat F = \mathrm{modReLU}(\widetilde F)$.
4. Inverse FFT: Reconstruct to the token domain by $n$0.

The end-to-end per-layer cost becomes $n$1—a significant improvement over the $n$2 cost of standard self-attention for large $n$3. Empirical ablations indicate that both the learnable spectral gate and the complex nonlinearity are critical for maintaining competitive accuracy with quadratic attention, with observed degradations of 1–2 points when they are removed (Fein-Ashley et al., 25 Feb 2025).

Variants such as Fourier Transformer (He et al., 2023) take a compression approach, running a DCT (via FFT) over hidden states, truncating high frequencies, and reconstructing via inverse DCT, which yields sparse, information-preserving representations. This enables efficient attention and feed-forward blocks by processing much shorter sequences post-truncation.

3. Hardware Acceleration for FFT-Based Models

Standard FFT or scan operations do not fully utilize modern GPU or accelerator architectures optimized for dense matrix multiplications (GEMM). To bridge this gap, two principal strategies have emerged:

• Spatial Dataflow Mapping and Specialized Interconnects: SSM-RDU (Ko et al., 29 Mar 2025) extends a baseline Reconfigurable Dataflow Unit with butterfly and crossbar networks inside each compute tile (PCU), enabling native spatial mapping of FFT and prefix-sum dataflows. These modifications incur less than 1% area and power overhead and allow near-ideal $n$4 and $n$5 scaling for Hyena and Mamba, respectively, yielding up to 5.95$n$6 kernel speedup over A100 GPUs when running long sequence models.
• Tensor Core Utilization via Matrix Decomposition: FlashFFTConv (Fu et al., 2023) reframes the Cooley–Tukey FFT as a series of large-dimension GEMMs using Kronecker factorization and block permutations, thus running on high-throughput tensor core hardware. By fusing FFT stages and eliminating off-chip memory bottlenecks, end-to-end convolutions achieve up to 4.4$n$7 training throughput improvements and 7.93$n$8 speedup on individual convolution kernels compared to standard batched FFT.

These hardware-aware methods ensure that FFT-based models fully harness contemporary accelerators' peak FLOP rates and memory hierarchies.

4. Advanced Algorithmic Extensions and Sparse FFT Variants

Several innovations extend beyond the vanilla application of FFT, targeting empirical bottlenecks and sequence model structure:

• Partial and Frequency-Sparse FFT Convolutions: FlashFFTConv introduces “partial” convolutions (tail pruning in time domain) and frequency-sparse convolutions (zeroing frequency blocks), directly skipping matrix multiplications in the decomposition. This yields further compute and memory reductions with no or minimal loss in accuracy, as evidenced by $n$9 to $\mathcal{O}(n^2)$0 additional speedup in specific settings (Fu et al., 2023).
• Structured Kernel Interpolation and Toeplitz Approximation: SKI-TNO (Moreno et al., 2023) splits bidirectional Toeplitz kernels into sparse (local convolution) and smooth (low-rank SKI) components, using FFT to accelerate the latter. For causal models, direct optimization in the frequency domain (modulating only the real part and reconstructing the imaginary via Hilbert transform) provides fast, causally correct mixing without explicit time-domain decay bias.
• DCT-Based Redundancy Truncation: In Fourier Transformer, DCT along sequence axes reveals that deep features concentrate in low frequencies. By truncating and reconstructing, the model operates on a reduced-length signal with negligible accuracy loss while fully conserving pre-trained weight compatibility.

5. Empirical Performance and Practical Impact

FFT-based acceleration strategies consistently produce strong empirical results across multiple benchmarks:

• SPECTRE/FFTNet (Fein-Ashley et al., 25 Feb 2025): Achieves 58.31% average accuracy on Long Range Arena (vs. 55.83% for Transformer), with 30–40% FLOP savings and better or equal accuracy on ImageNet-1K. Latency benchmarks demonstrate wall-clock speed gains even at small batch sizes.
• FlashFFTConv (Fu et al., 2023): Delivers 4.4$\mathcal{O}(n^2)$1 end-to-end throughput gains and up to $\mathcal{O}(n^2)$2 per-kernel speedups on H100 GPUs for long-sequence Hyena and DNA models, enabling training with sequence lengths up to 4 million bases or large vision tasks (Path-512 and Path-X). Hardware FLOP utilization increases from ~20% (standard FFT) to 62%, crossing the threshold demanded for state-of-the-art generative and classification tasks.
• Fourier Transformer (He et al., 2023): Surpasses previous models on LRA (67.54% average accuracy), with $\mathcal{O}(n^2)$3 speed and $\mathcal{O}(n^2)$4 peak memory at $\mathcal{O}(n^2)$5k context. Maintains or improves performance in seq-to-seq and summarization by fine-tuning inherited BART weights.
• SSM-RDU (Ko et al., 29 Mar 2025): Demonstrates $\mathcal{O}(n^2)$6 accelerator speedup for Hyena and $\mathcal{O}(n^2)$7 for Mamba over A100, with negligible (<1%) area/power overhead for custom butterfly/scan interconnects.

6. Theoretical Analysis, Trade-offs, and Limitations

While FFT-based approaches provide substantial practical acceleration, several technical caveats and expressivity trade-offs exist:

• Expressivity vs. Efficiency: Spectral mixing or convolutional layers mediated by FFT are less flexible for modeling arbitrarily fine-grained pairwise or position-dependent interactions compared to full dot-product attention. Ablation in SPECTRE/FFTNet shows modest accuracy drops when removing complex-valued nonlinearity or spectral gating, indicating that global filters, while powerful, incompletely replicate all attention behaviors (Fein-Ashley et al., 25 Feb 2025).
• Memory Usage: FFT-based models generally require $\mathcal{O}(n^2)$8 complex activation buffers. However, this is still a net improvement over attention's $\mathcal{O}(n^2)$9 memory when $\mathcal{O}(n\log n)$0 is large.
• Numerical Stability: Norm preservation properties (e.g., via Parseval's theorem) are beneficial. Nonlinear activations such as modReLU may zero frequencies but do not lead to numerical explosions in practice.
• Autoregressive and Causal Operations: Prefix-FFT caching and incremental frequency updates for decoder efficiency are not universally addressed; such extensions are highlighted as open directions in several works (Fein-Ashley et al., 25 Feb 2025, He et al., 2023).

7. Outlook and Future Directions

FFT-based acceleration underpins a robust direction in efficient long-sequence modeling, extending from high-level model design to hardware co-optimization. The cross-disciplinary synergy between algorithmic innovations (e.g., spectral token mixing, DCT truncation, partial/sparse FFTs, Hilbert-enforced causality) and hardware-aware implementations via custom dataflows or tensor core decompositions continues to expand the feasible regime of model and context sizes.

Open areas include:

• Development of incremental, cache-friendly FFTs for streaming or autoregressive generation (Fein-Ashley et al., 25 Feb 2025, He et al., 2023).
• Adaptive masking, frequency selection, or hybrid local-global spectral methods.
• Broader integration with pretrained foundation models and compatibility with diverse sequence modalities (audio, vision, genomics).
• Further accelerator enhancements with minimal hardware overhead (Ko et al., 29 Mar 2025), potentially extending to broader families of structured transforms.

Consolidation of these principles is expected to be central in next-generation architectures for high-throughput, long-context modeling across scientific and industrial applications.