Reverse Fourier Attention (RFA)

• Reverse Fourier Attention is a method that applies random Fourier features to approximate kernelized self-attention, significantly reducing computational costs.
• It achieves linear time and memory complexity by replacing the quadratic softmax attention with efficient feature map approximations, improving scalability.
• RFA forms the basis for extended models such as RMFA and Macformer, which demonstrate effectiveness in benchmark tasks like Long Range Arena.

Reverse Fourier Attention (RFA) refers to a class of efficient attention mechanisms leveraging random Fourier feature (RFF) methods to approximate kernelized attention functions, primarily the softmax kernel used in standard Transformers. By utilizing RFFs, RFA achieves linear time and memory complexity relative to sequence length, in contrast to the quadratic costs of classical self-attention. RFA has become foundational for a spectrum of scalable attention architectures, including those using alternative feature construction schemes and kernelized approximations (Guo et al., 2024).

1. Mathematical Foundations of Random Fourier Feature Attention

RFA is grounded in kernel approximation theory, specifically the random Fourier feature construction for shift-invariant, positive definite kernels as articulated by Bochner’s theorem. In the transformer context, the softmax attention kernel is $k(x, y) = \exp(x^\top y / \sqrt{d})$. For generic continuous, shift-invariant kernels $k(\delta)$, Bochner's theorem ensures a real-valued feature embedding $\phi(x)$ such that $k(x, y) \approx \phi(x)^\top \phi(y)$ via Monte Carlo sampling:

• Sample $w \sim p(w)$, $b \sim \text{Uniform}[0, 2\pi]$,
• Define $$\phi(x) = \sqrt{2/m} [\cos(w_1^\top x + b_1), ..., \cos(w_m^\top x + b_m)]^\top$$,
• Approximate $k(x, y)$ by Monte Carlo: $k(x, y) \approx \phi(x)^\top \phi(y)$.

This leads to an unbiased attention approximation for softmax kernels:

$$\exp(x^\top y / \sqrt{d}) \approx \phi(x)^\top \phi(y)$$

The RFA approach enables efficient computation through feature maps, circumventing the need to materialize the full $Q K^\top$ attention matrix.

2. RFA Algorithmic Structure and Complexity

The Random Fourier Feature Attention procedure utilizes precomputed random projections and phases to embed $Q$ and $K$ matrices into low-dimensional feature spaces. The workflow proceeds as follows (for single-head attention):

1. Compute $\phi(Q)$ and $\phi(K)$ row-wise ($\mathbb{R}^{n \times m}$).
2. Compute normalization and aggregation:
   • $S = \phi(K)^\top V$
   • $z = \phi(K)^\top 1_n$
3. Form output:
   • $Out = \phi(Q) S / (\phi(Q) z)$ (element-wise division).

This algorithm has $O(n m d)$ time and $O(n m + m d)$ space complexity, representing a reduction to linear overhead with respect to sequence length $n$ given $m \ll n$.

3. Extensions Beyond RFA: Random Maclaurin Feature Attention and Kernel Choices

RFA serves as the basis for broader mechanisms whereby generic dot-product kernels possessing non-negative Maclaurin expansions can be approximated. Macformer introduces Random Maclaurin Feature Attention (RMFA), leveraging features constructed by sampling degree $N \sim \text{Geometric}(p)$ and independent Rademacher vectors $\omega_1, ..., \omega_N$:

$$\phi(x) = \sqrt{a_N p^{N+1}} \cdot (\omega_1^\top x) \cdot ... \cdot (\omega_N^\top x)$$

This scheme generalizes RFA to arbitrary dot-product kernels $K(t) = \sum_{N=0}^\infty a_N t^N$ with $a_N \geq 0$ (Guo et al., 2024). RMFA is unbiased, satisfying $$\mathbb{E}_{N, \omega} [\phi(x)\phi(y)] = K(x^\top y)$$, and supports approximation error bounds scaling exponentially with feature number $M$.

4. Normalization Strategies: pre–post Scaling Batch Normalization

For kernels with restricted input domains (e.g., $K_{inv}$, $K_{log}$), Macformer introduces pre–post Scaling Batch Normalization (ppSBN) to maintain norm constraints:

1. Batch normalize $Q$, $K$: $\hat{Q}$, $\hat{K}$
2. Project to unit norm: $\tilde{Q}$, $\tilde{K}$
3. Apply RMFA: $$\text{att} = \text{RMFA\_exp}(\tilde{Q}, \tilde{K}, V)$$
4. Rescale with learnable parameters $\gamma, \beta$: $\text{Output} = (\gamma \cdot \text{att})^\beta$

This mechanism regularizes RMFA output for kernels sensitive to input scaling, restoring appropriate magnitude via learned rescaling.

5. Empirical Evaluation

Benchmarks on the Long Range Arena (LRA) evaluate both RFA and Macformer RMFA architectures against baseline Transformer models for test, ListOps, and retrieval tasks (Guo et al., 2024). Comparative metrics illustrate computational and accuracy impacts:

Task	Model	Time	Mem	Accuracy
Text	Transformer	1.00	1.00	63.31
	Transformer_RFA	0.78	1.37	65.22
	Macformer_exp	0.31	1.34	64.06
ListOps	Transformer	1.00	1.00	37.30
	Transformer_RFA	0.84	1.36	37.15
	Macformer_trigh	0.65	2.56	38.36
Retrieval	Transformer	1.00	1.00	75.04
	Transformer_RFA	0.90	2.39	77.84
	Macformer_log	0.35	1.89	70.73

A plausible implication is that RFA and its RMFA-derived variants can achieve time reductions of 30–70% compared to quadratic attention, often without degradation in accuracy. Different kernel choices may confer task-specific advantages, as evidenced by Macformer_trigh’s performance on Listops.

6. Context, Related Methods, and Theoretical Implications

RFA has anchored a family of scalable kernelized attention architectures for long-context sequence modeling. By leveraging spectral methods for kernel approximation, these approaches sidestep explicit calculation of full attention matrices—a paradigm further developed in Macformer via random Maclaurin features. While RFA is strictly for shift-invariant kernels approximable via RFF, RMFA generalizes to arbitrary dot-product forms subject to expansion constraints. These developments relate fundamentally to the work of Peng et al. (2021) on Random Feature Attention and Kar & Karnick (2012) on random feature maps for dot-product kernels.

This suggests ongoing theoretical work is focused on kernel selection, bias-variance trade-offs in feature approximation, and regularization strategies required for nonstandard kernels. The use of ppSBN in Macformer is indicative of such adaptation requirements when deploying generalized kernel attention at scale.

7. Limitations and Prospects

While RFA and RMFA-derived models exhibit improved computational profiles and stable accuracy for a range of benchmarks, their efficacy depends on kernel properties, choice of feature dimension $m$, and task structure. Kernel selection must be tuned to individual modalities to achieve optimal performance. Moreover, although error bounds are theoretically sound, empirical variance may arise in practical deployments for extremely long contexts or under distributional shift.

A plausible implication is that ongoing developments—including spectral gating, advanced caching (e.g., Prefix-FFT cache), and hybrid wavelet modules (as referenced in other kernelized FFT-based methods)—may further enhance the scalability and representational power of Fourier-inspired attention mechanisms.

References

• Guo et al., "Macformer: Transformer with Random Maclaurin Feature Attention" (Guo et al., 2024).
• Peng et al., "Random Feature Attention," 2021.
• Kar & Karnick, "Random feature maps for dot product kernels," AISTATS 2012.