---
title: 'Fourier Features: Theory & Applications'
url: https://www.emergentmind.com/topics/fourier-features
type: topic
---

# Fourier Features: Theory & Applications

Searching arXiv for recent and foundational work on Fourier features to ground the encyclopedia entry.
arxiv_search(query="Fourier features positional encoding random Fourier features neural tangent kernel Gaussian processes reinforcement learning attribution", max_results=10)
Fourier features are explicit sinusoidal embeddings that map an input coordinate, state, or feature vector into a basis of sines and cosines, typically before application of a downstream predictor such as a multilayer perceptron, kernel model, Gaussian process approximation, or sequence model. In modern machine learning, the term encompasses several related constructions: fixed random Fourier features derived from Bochner-type spectral representations of stationary kernels; deterministic or quadrature-based Fourier features for improved approximation accuracy; learned Fourier bases whose frequencies are trained jointly with the network; and deep Fourier-feature architectures that insert sine–cosine pairs throughout the network rather than only at the input. Across these settings, Fourier features are used to reshape inductive bias in frequency space, often to mitigate spectral bias, alter effective kernel bandwidth, improve approximation of oscillatory structure, or furnish scalable low-rank surrogates for otherwise expensive kernel methods [2006.10739], [2109.01545], [2310.14544], [2112.03257].

## 1. Definition and canonical constructions

A standard Fourier feature map replaces a raw input \(v\in\mathbb{R}^d\) by a higher-dimensional sinusoidal embedding. In one influential coordinate-based formulation, the map is written as
\[
\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],
\]
where \(\mathbf b_j\) are frequencies and \(a_j\) are amplitudes [2006.10739]. This construction subsumes positional encodings based on fixed log-spaced frequencies as well as random Fourier features obtained by sampling frequencies from a spectral distribution.

In reinforcement learning, a learned variant embeds the input \(x\) by
\[
F_B(x) = \sin(2\pi Bx)\ \Vert\ \cos(2\pi Bx)\ \Vert\ x,
\]
with a trainable basis matrix \(B\), after which a standard MLP or CNN stack is applied [2112.03257]. The concatenation with the original input preserves raw coordinates alongside the sinusoidal basis.

A different line of work applies Fourier features not only at the input but throughout the depth of the network. In that setting, each pre-activation \(z_l\) is mapped elementwise as
\[
h_l = [\sin(z_l), \cos(z_l)],
\]
so that every hidden layer emits a paired sine–cosine representation [2410.20634]. This “deep Fourier features” construction differs materially from shallow positional encodings because it changes the internal trainability and optimization geometry of the network, not merely its input parameterization.

Fourier features also appear in explicit kernel approximations. For stationary kernels, the feature map is designed so that
\[
k(x,x') \approx \langle z(x), z(x')\rangle,
\]
turning kernel learning into linear learning in feature space [2109.10623], [2109.01545]. In Gaussian process regression, the same idea yields low-rank covariance approximations with tractable training and prediction [2310.14544].

## 2. Spectral viewpoint and kernel-theoretic basis

The theoretical basis for random Fourier features is the spectral representation of stationary kernels. For a stationary kernel \(k(x,x')=k(x-x')\), Bochner’s theorem permits the representation
\[
k(x-x')=\int p(\omega)\,e^{i\omega^\top(x-x')}\,d\omega,
\]
so a finite-dimensional approximation can be formed by replacing the integral with sampled or deterministic frequencies [2109.01545]. In the standard random construction, frequencies are sampled from the spectral density and assembled into an explicit map such as
\[
z(x)=\sqrt{\frac{2}{M}}\big[\cos(\omega_1^\top x+b_1),\dots,\cos(\omega_M^\top x+b_M)\big],
\]
yielding a Monte Carlo approximation whose error decays at the usual \(\mathcal{O}(M^{-1/2})\) rate [2109.01545].

This kernel perspective remains central even in neural-network applications. For coordinate MLPs, passing the input through a Fourier map changes the effective neural tangent kernel. The composed model behaves with a stationary kernel
\[
k_\gamma(v_1, v_2) = \gamma(v_1)^\top \gamma(v_2)
= \sum_{j=1}^m a_j^2 \cos\!\left(2\pi \mathbf b_j^\top (v_1-v_2)\right)
= h_\gamma(v_1-v_2),
\]
so the inductive bias changes from a dot-product geometry to a shift-invariant one [2006.10739]. This is a central reason Fourier features alter what functions are learned quickly and which frequencies are favored during optimization.

In classification, random Fourier feature approximations inherit statistical guarantees from the underlying kernel method. For Lipschitz losses such as support vector machine and logistic regression, random Fourier feature classifiers can attain an \(O(1/\sqrt n)\) learning rate with only \(\Omega(\sqrt n \log n)\) features in the benign case \(r=1\), improving on earlier \(\Omega(n)\) feature requirements; under Massart’s low noise assumption, fast \(O(1/n)\) rates are also possible [2109.10623]. This places Fourier features at the intersection of kernel approximation and finite-sample statistical learning theory.

## 3. Spectral bias, frequency control, and optimization

A major reason Fourier features became prominent in deep learning is their relation to spectral bias. Standard coordinate-based MLPs preferentially learn low-frequency structure and are slow to fit high-frequency detail. In low-dimensional regression tasks, this limitation can be severe enough that plain MLPs fail to recover sharp spatial structure both in theory and in practice [2006.10739]. Fourier features mitigate this by furnishing the network with a basis already aligned with oscillatory components.

From the neural tangent kernel viewpoint, learning speed is governed by kernel eigenvalues. In one formulation, the training dynamics are approximated by
\[
Q^\top (\mathbf{\hat{y}^{(t)}}_{\mathrm{train}} - \mathbf y) \approx -e^{-\eta \Lambda t} Q^\top \mathbf y,
\]
so components corresponding to small eigenvalues decay slowly [2006.10739]. Fourier embeddings alter the spectrum of the effective kernel and thereby accelerate fitting of higher-frequency modes.

In reinforcement learning, this spectral argument is sharpened into a functional regularization interpretation. For learned Fourier features analyzed in the infinite-width NTK regime, the residual under gradient flow evolves as
\[
f_{\theta_t}(x) - y = e^{-\eta K t}\,(f_{\theta_0}(x)-y),
\]
with learning rates determined by the eigenvalues of the NTK matrix \(K\) [2112.03257]. For a two-layer learned-Fourier-feature model on the sphere, the kernel takes the form
\[
k(x, x') = \left(1 + x^\top x'\right)\exp\!\left\{-\frac{\sigma^2}{2}\|x-x'\|_2^2\right\},
\]
equivalently
\[
k(x,x') = \left(1+\cos\theta\right)\exp\!\left\{\sigma^2(\cos\theta - 1)\right\},
\]
making the role of the Fourier initialization variance \(\sigma^2\) explicit [2112.03257]. Small \(\sigma\) yields broader, smoother kernels and suppresses high frequencies; large \(\sigma\) yields more localized kernels and faster learning of sharper structure. The paper interprets this as direct control over underfitting versus overfitting of frequencies, particularly relevant when Bellman targets are noisy [2112.03257].

A related but distinct trainability argument appears in continual learning. Deep Fourier features, defined as concatenated sine and cosine activations in every layer, are motivated by the observation that for any pre-activation \(z\), at least one of \(\sin(z)\) or \(\cos(z)\) is locally well approximated by a linear function on an interval of width \(\pi/2\) [2410.20634]. This yields a dynamic balance between nonlinearity and an approximate linear pathway, which the authors connect to preservation of plasticity across non-stationary tasks [2410.20634].

## 4. Variants and methodological families

Fourier features now comprise several distinct methodological families rather than a single technique.

### Fixed random Fourier features

The classical random Fourier feature method samples frequencies independently from a spectral density and uses a finite sinusoidal map to approximate a stationary kernel [2109.01545]. Its chief advantages are simplicity, compatibility with linear solvers, and scalability relative to exact kernel methods. Its main limitation is Monte Carlo error and the resulting need for many features in difficult regimes [2109.01545], [2109.10623].

### Deterministic and quadrature-based features

To reduce stochastic approximation error, deterministic constructions replace Monte Carlo sampling by quadrature or structured spectral bases. Deterministic Fourier features can achieve exponentially decaying approximation error in the number of basis functions under suitable one-dimensional smoothness and domain assumptions, but naïve tensor-product extensions suffer from the curse of dimensionality [2109.01545]. Trigonometric Quadrature Fourier Features (TQFF) further adapt quadrature to oscillatory Fourier integrals by constructing a rule exact for trigonometric polynomials rather than ordinary polynomials, improving approximation accuracy and uncertainty calibration in Gaussian process regression with fewer features over a broad range of length-scales [2310.14544].

### Learned Fourier features

In learned Fourier features, the frequency matrix is optimized by backpropagation rather than fixed a priori [2112.03257]. This allows the embedding itself to adapt to the task, while retaining an interpretable frequency-control parameter through the initial variance of the basis. Learned bases are particularly relevant in high-dimensional reinforcement-learning settings where fixed random features can be inadequate [2112.03257].

### Deep Fourier features

Deep Fourier features replace conventional hidden activations with sine–cosine pairs at every layer [2410.20634]. This is not merely a richer basis for the input; it redefines the hidden state space and, according to the cited analysis, approximately embeds a deep linear network inside a nonlinear architecture. The aim is not primarily representation of high-frequency inputs, but sustained trainability in continual learning [2410.20634].

### Quantized and filtered Fourier embeddings

In implicit neural representations, Quantized Fourier Features (QFF) quantize Fourier-feature values into bins with learnable embeddings, combining periodic multiresolution structure with explicit local parameterization [2212.00914]. A separate line robustifies Fourier embeddings by inserting a bias-free MLP filter acting multiplicatively on the Fourier channels,
\[
\tilde y(v)=f_a(y(v)) \odot y(v),
\]
to suppress unnecessary frequencies or create sum-and-difference frequencies through channel interactions [2502.05482]. This suggests a shift from treating Fourier features as fixed preprocessing to treating them as objects of adaptive frequency modulation.

### Nonstationary and structured-domain extensions

Fourier features have also been generalized beyond stationary Euclidean kernels. Regular Fourier features for harmonizable Gaussian processes discretize the spectral representation directly on a regular frequency grid and preserve correlations among spectral weights, avoiding the probability-density interpretation used in stationary random Fourier features [2602.23006]. Random Fourier Signature Features lift random Fourier approximations into the signature-kernel framework for sequences, producing unbiased explicit features for ordered, variable-length time series [2311.12214]. These developments indicate that “Fourier features” increasingly denotes a general spectral-design paradigm rather than only Rahimi–Recht-style random features.

## 5. Major application domains

### Coordinate-based neural representations

The modern revival of Fourier features is closely associated with low-dimensional coordinate regression in vision and graphics. Injecting sinusoidal embeddings into coordinate MLPs markedly improves image regression, 3D shape regression, CT reconstruction, MRI reconstruction, and NeRF-style inverse rendering [2006.10739]. The improvements are explained by the ability of the transformed model to learn high-frequency functions in low-dimensional domains that plain MLPs oversmooth [2006.10739]. Positional encodings used in neural radiance fields are a structured special case of this broader Fourier-feature framework.

### Reinforcement learning

Learned Fourier features have been used to improve sample efficiency and stability in off-policy deep reinforcement learning [2112.03257]. In experiments on DeepMind Control Suite tasks with SAC, the architecture beats MLP baselines in 6 out of 8 state-based environments and yields clear gains on image-based benchmarks with SAC + RAD [2112.03257]. The reported gains concentrate in the critic rather than the policy, consistent with the hypothesis that frequency-space regularization is especially valuable where Bellman bootstrapping introduces noise [2112.03257].

### Continual learning

In continual supervised learning, deep Fourier features improve performance under label noise, class-incremental learning, and pixel permutations on CIFAR10, CIFAR100, and tiny-ImageNet by replacing ReLU activations with sine–cosine pairs [2410.20634]. The motivating claim is that linear function approximation and a special case of deep linear networks do not suffer from loss of plasticity, and deep Fourier features inherit part of this trainability while retaining nonlinear expressivity [2410.20634].

### Gaussian processes and scalable kernel methods

Fourier features remain central to scalable Gaussian process inference. Random, deterministic, quadrature, and RKHS-based Fourier features all reduce GP complexity by replacing dense kernel matrices with low-rank explicit feature maps [2310.14544], [2407.01856], [2602.23006]. In compositional and deep GP models, ODE-based RKHS Fourier response features modulate amplitude and phase through convolution with a Green’s function, extending fixed sinusoidal bases to dynamical settings [2407.01856].

### Periodic and high-dimensional kernels

For multidimensional periodic kernels, Index Set Fourier Series Features approximate the kernel using deterministic multivariate Fourier series restricted to sparse index sets, outperforming random Fourier features on periodic regression problems and kernel reconstruction [1805.04982]. This setting emphasizes a less publicized but historically central role of Fourier features: direct exploitation of harmonic structure in periodic covariance models.

### Imitation learning and 3D perception

Point-cloud imitation learning has recently adopted axis-aligned NeRF-style Fourier features on Cartesian coordinates to improve high-precision robotic manipulation [2606.12334]. The method maps each coordinate independently across multiple wavelengths and improves performance across several point-cloud encoders on RoboCasa, ManiSkill3, and real-robot tasks [2606.12334]. The stated rationale is again spectral bias: raw Cartesian features are “slow-moving,” whereas Fourier mappings make fine geometric distinctions more directly available to the policy network [2606.12334].

### Explainable AI and attribution

Fourier features have also become an object of attribution analysis. A recent method scores Fourier coefficients using an error-response construction
\[
Score = \left|\mathcal{F}(X[n]\cdot \nabla X[n])\right| - \left|\mathcal{F}(X[n])\right|,
\]
arguing that deletion in Fourier space corresponds to genuine removal of signal components rather than introduction of pixel-space artifacts [2504.02016]. In ImageNet experiments with Vision Transformers, only \(8\%\) of Fourier features are required to maintain the original predictions for \(80\%\) of samples [2504.02016]. This suggests that Fourier features may function not only as engineered inputs but also as an explanatory basis for model behavior.

## 6. Theoretical interpretations and broader significance

Fourier features admit several complementary interpretations, none of which exhaust the concept.

First, they are explicit basis expansions that convert nonlinear function approximation into learning over a richer linear span of oscillatory atoms. In the simplest view, they expose high-frequency directions to a model that would otherwise discover them only slowly [2006.10739].

Second, they are kernel approximators. Whether frequencies are random, quadrature-based, or learned, the feature map induces an approximate kernel, and much of the theory concerns approximation error, effective dimension, and generalization [2109.01545], [2109.10623], [2310.14544].

Third, they are mechanisms for frequency-space regularization. The learned-Fourier-feature reinforcement-learning analysis makes this explicit by tying the initial basis variance to the underfitting–overfitting trade-off across frequencies [2112.03257]. This perspective differs from ordinary weight decay because it regulates the learned function rather than only parameter norms.

Fourth, they are tied to invariance and harmonic analysis more broadly. For networks invariant to a finite group, weights can be shown to recover the Fourier transform on that group, with matrix-valued irreducible unitary representations replacing scalar Fourier modes in the non-commutative case [2312.08550]. This result places Fourier features within a larger algebraic theory of invariant representations rather than only the Euclidean sinusoidal setting.

A common misconception is that Fourier features are synonymous with positional encoding. Positional encodings are one important instance, especially in neural fields, but the literature also includes random and deterministic kernel approximations, learned frequency matrices, deep sine–cosine activations, response features shaped by ODE dynamics, and nonstationary spectral discretizations [2006.10739], [2112.03257], [2410.20634], [2407.01856], [2602.23006].

Another misconception is that Fourier features invariably improve performance. The literature instead presents a recurring bias–variance or smoothness–detail trade-off. Too narrow a frequency spectrum induces underfitting and oversmoothing; too wide a spectrum can produce aliasing, overfitting, or noise sensitivity [2006.10739], [2112.03257], [2310.14544], [2502.05482]. The choice of scale, distribution, or adaptive mechanism is therefore integral to the method rather than a peripheral hyperparameter.

## 7. Limitations, controversies, and open directions

Several limitations recur across the literature. Random Fourier features converge slowly by Monte Carlo sampling, motivating deterministic and adaptive alternatives [2109.01545], [2310.14544]. Deterministic tensor-product features can suffer from the curse of dimensionality unless coupled with low-rank tensor structure or sparse index sets [2109.01545], [1805.04982]. In neural fields, fixed Fourier embeddings may introduce noisy outputs or unstable reconstructions when the sampled basis is mismatched to the target spectrum [2502.05482]. In high-dimensional settings, fixed random features can be inferior to learned or structurally adapted bases [2112.03257].

There is also no single answer to the question of whether Fourier features should be fixed or learned. One line of work reports that jointly training the Fourier frequencies in low-dimensional regression does not help much because the frequencies move little from initialization [2006.10739]. Another shows that training the basis matrix is important in high-dimensional reinforcement learning and that fixed random Fourier features work worse as dimensionality increases [2112.03257]. This suggests that the utility of learning the basis is strongly regime-dependent.

A further tension concerns robustness versus expressivity. Richer Fourier embeddings can recover fine detail, but they may also amplify high-frequency noise or degrade calibration. This tension motivates filtered embeddings in INRs [2502.05482], functional regularization in reinforcement learning [2112.03257], and trigonometric quadrature constructions in Gaussian processes [2310.14544]. A plausible implication is that the future of Fourier features lies less in simply adding more frequencies than in designing task-aligned spectral priors and adaptive frequency controls.

Recent work points toward several extensions. Nonstationary Gaussian-process models now use regular-grid spectral discretizations that preserve cross-frequency correlations [2602.23006]. Compositional GP models employ ODE-modulated RKHS Fourier response features with adaptive amplitude and phase [2407.01856]. Attribution methods treat Fourier coefficients as the primary explanatory units for model decisions [2504.02016]. Group-invariance theory links learned representations to generalized Fourier transforms on finite groups [2312.08550]. Taken together, these developments suggest that Fourier features have evolved from a practical trick for positional encoding into a unifying spectral language spanning approximation theory, kernel methods, representation learning, reinforcement learning, continual learning, and explainable AI.

Source: https://www.emergentmind.com/topics/fourier-features