FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

Abstract: FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val=0.608 (Delta=+0.867); and four learned frequencies spanning paragraph to word scale achieve val=0.309 (Delta=+1.166), a 79% reduction over standard dot-product attention. The single-frequency result is confirmed across three random seeds (mean val=0.236, std=0.019). The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales. The gain is specific to spectral preprocessing: random orthogonal and non-orthogonal projections of Q/K produce no measurable improvement, suggesting the benefit comes from global frequency-domain mixing rather than metric distortion. All results are verified by a shuffled-validation diagnostic against positional leakage. Causal filters (Gaussian, Mexican Hat, Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT kernel is structurally non-causal, coupling every position to future tokens. This defines an architectural boundary between bilateral spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation (companion paper MorletQK). This work is architecturally distinct from FNet (Lee-Thorp et al., 2021), which replaces attention with Fourier mixing of token embeddings. Here, spectral preprocessing applies only to Q/K projections while the full attention score structure is preserved.

• The paper demonstrates that applying FFT-based spectral filtering to query-key projections reduces validation loss by up to 79% compared to standard attention.
• It introduces learned frequency-selective filters that capture multi-scale linguistic structures, revealing a clear spectral ‘sweet spot’ at the paragraph scale.
• The study highlights that maintaining phase coherence is critical to achieve leakage-free and causally sound transformer attention mechanisms.

FourierQK: Spectral Preprocessing of Query-Key Projections in Transformer Attention

Introduction and Motivation

The paper "FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention" (2607.07478) systematically investigates whether frequency-domain transformations of query (Q) and key (K) projections can enhance the performance of transformer attention. Standard attention involves dot products between learned Q/K projections at each token position, capturing local similarity. This research instead introduces frequency-domain filters via the Fast Fourier Transform (FFT) applied to Q/K matrices before score computation, enabling global mixing of sequence information.

Motivated by prior work that established energy and phase as critical signals in transformer representations, the author hypothesizes that making Q/K projections sensitive to specific spectral components may provide inductive biases aligned with linguistic structure. The distinction from FNet [lee2021fnet] and other spectral techniques is clear: FourierQK preserves the attention score matrix and only preprocesses Q/K.

Spectral Attention: Architecture and Variants

Spectral Attention (SA) transforms $q, k \in \mathbb{R}^{T \times d}$ by (i) performing a DFT along the sequence dimension, (ii) applying a frequency-domain filter $\phi(\omega)$, and (iii) reconstructing filtered Q/K representations and computing standard dot-product attention. The procedure yields

$$\tilde{q}(b) = \mathcal{F}^{-1}\big[ \hat{q}(\omega) \phi(\omega) \big](b)$$

with analogous computation for $\tilde{k}$, followed by score computation. Crucially, the filter can be (a) fixed random, (b) single or multiple learned frequency selectors, or (c) causal time-domain convolutions (Gaussian/Morlet/Mexican Hat).

These options test whether general spectral mixing, specific frequency selectivity, or causal convolutions contribute to attention improvements.

Figure 1: Summary of FourierQK results, illustrating learning dynamics, phase impact, frequency scale emergence, and evaluation of bug-fixed vs. buggy implementations.

Numerical Results and Analysis

Gains from Spectral Preprocessing

The application of FFT-based filtering to Q/K projections was found to yield strong improvements in validation loss on the TinyShakespeare character-level modeling benchmark. Key quantitative results include:

• Random fixed filter: Val = 1.031 ($\Delta = +0.443$ over baseline)
• Single learned frequency: Val = 0.608 ($\Delta = +0.867$)
• Multi-frequency (4 learned frequencies): Val = 0.309 ($\Delta = +1.166$), 79% reduction over standard dot-product attention

Ablation and robustness studies confirm these gains are reproducible and not seed-specific. Multi-seed experiments indicate that the learned frequency selectors reliably converge to similar values corresponding to interpretable linguistic scales (paragraph, phrase, word).

Control and Causal Variants

By contrast, non-spectral linear projections (random orthogonal or non-orthogonal) provide no improvement, confirming that the gain is intrinsically spectral. Causal time-domain filters—Gaussian, MexHat, Morlet—consistently underperform standard attention at character scale.

Figure 2: Theiler-style surrogate analysis quantifies the essential contribution of phase coherence over amplitude in spectral attention, with phase randomization causing dramatic performance drops.

Spectral Scale and Phase Sensitivity

Analysis of which frequencies contribute maximally revealed a non-monotonic structure: a clear "sweet spot" at the paragraph scale ($\sim$64 tokens/cycle). Very low and very high frequencies are less informative, while paragraph- and subparagraph-scale frequencies dominate the improvement. Multi-frequency models discover a near-dyadic hierarchy with periods tightly matching linguistic units.

Critically, phase coherence between Q and K projections is essential. Phase randomization (Theiler surrogate testing) worsens performance well below baseline, while amplitude-only spectral information is not beneficial. The attention mechanism's gain resides in its ability to exploit temporally recurring, phase-coherent patterns in the sequence.

Causality, Spectral Leakage, and Attention Mechanism Integrity

A substantial part of the paper addresses the integrity of spectral attention mechanisms, particularly the risks posed by spectral leakage through irfft-based position reconstruction. Bilateral FFT kernels are shown to be non-causal and allow future token information to propagate backward. Diagnostics based on shuffled validation data (the "shuffled gap") are crucial for ruling out leakage: FFT-leaking models exhibit smaller gaps, whereas the proposed FourierQK retains a large shuffled gap consistent with genuine sequence learning.

Notably, only operations that collapse the frequency axis (rather than reconstructing the sequence) are shown to be leakage-free and genuinely causal in the attention mechanism context.

Implications for Language Modeling and Transformer Architecture

The demonstrated improvements using spectral preprocessing of Q/K projections suggest that meaningful global structure at language-relevant scales is present even in character-level data, and can be efficiently harnessed by equipping transformers with spectral inductive biases. The multi-resolution frequency selectors autonomously discover a linguistic hierarchy without explicit supervision.

The failure of local causal filters at character scale identifies an architectural limitation and emphasizes the necessity of global sequence mixing for certain language modeling tasks. This limitation is conjectured to diminish at word-scale tokenization, paving the way for extensions to subword or word-level predictors.

Theoretical and Practical Implications

Theoretically, these findings challenge the sufficiency of purely local or content-based dot-product mechanisms in transformer attention. Spectral structure and phase information are shown to be not just complementary but essential to achieving optimal sequence modeling performance. The clear partitioning of spectral attention mechanisms into those that are leakage-prone and those that are not provides guidance for the rigorous development of future architectures.

For practitioners, incorporating learned frequency-domain Q/K preprocessing can be implemented as a drop-in enhancement in transformer attention without disturbing the causal masking or value aggregation pipeline. Practical scaling to longer contexts and more complex domains, as well as the design of learnable multi-scale filters, are indicated as immediate next steps.

Future Directions

The paper identifies several concrete avenues for future research:

• Extension of spectral attention to word/subword level tokenization, where kernel support and spectral aliasing issues are mitigated.
• Learning frequency selector width per head to dynamically trade off spectral and temporal resolution.
• Incorporating admissible wavelet-based frequency collapse, potentially combining the strengths of multi-scale attention and spectral selectivity.
• Exploring non-learned vs. dynamically learned spectral filters and their relationship to linguistic structure in diverse domains.
• Rigorous evaluation in larger-scale and multilingual settings.

Conclusion

This work establishes that FFT-based spectral preprocessing of learned query and key projections enables exceptionally strong improvements in transformer attention for language modeling, with best-case reductions exceeding 75% in validation loss over baseline attention. The improvements are robust across random seeds and primarily owed to the exploitation of phase-coherent, multi-scale structures within the token sequence. The investigation demarcates the boundaries between globally mixing, leakage-prone, and truly causal spectral mechanisms, and provides a principled foundation for further advances in frequency-aware transformer design.