Fourier Features: Theory & Applications

• Fourier Features are explicit sinusoidal embeddings that map input data into a high-dimensional frequency space for efficient kernel approximation.
• They mitigate spectral bias by enabling neural networks to learn high-frequency details and accelerate convergence in regression tasks.
• Multiple constructions exist—fixed random, deterministic, learned, and deep Fourier features—each balancing expressivity with computational efficiency.

Searching arXiv for recent and foundational work on Fourier features to ground the encyclopedia entry. arxiv_search(query="Fourier features positional encoding random Fourier features neural tangent kernel Gaussian processes reinforcement learning attribution", max_results=10) Fourier features are explicit sinusoidal embeddings that map an input coordinate, state, or feature vector into a basis of sines and cosines, typically before application of a downstream predictor such as a multilayer perceptron, kernel model, Gaussian process approximation, or sequence model. In modern machine learning, the term encompasses several related constructions: fixed random Fourier features derived from Bochner-type spectral representations of stationary kernels; deterministic or quadrature-based Fourier features for improved approximation accuracy; learned Fourier bases whose frequencies are trained jointly with the network; and deep Fourier-feature architectures that insert sine–cosine pairs throughout the network rather than only at the input. Across these settings, Fourier features are used to reshape inductive bias in frequency space, often to mitigate spectral bias, alter effective kernel bandwidth, improve approximation of oscillatory structure, or furnish scalable low-rank surrogates for otherwise expensive kernel methods (Tancik et al., 2020, Wesel et al., 2021, Li et al., 2023, Li et al., 2021).

1. Definition and canonical constructions

A standard Fourier feature map replaces a raw input $v\in\mathbb{R}^d$ by a higher-dimensional sinusoidal embedding. In one influential coordinate-based formulation, the map is written as

$$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$

where $\mathbf b_j$ are frequencies and $a_j$ are amplitudes (Tancik et al., 2020). This construction subsumes positional encodings based on fixed log-spaced frequencies as well as random Fourier features obtained by sampling frequencies from a spectral distribution.

In reinforcement learning, a learned variant embeds the input $x$ by

$$F_B(x) = \sin(2\pi Bx)\ \Vert\ \cos(2\pi Bx)\ \Vert\ x,$$

with a trainable basis matrix $B$, after which a standard MLP or CNN stack is applied (Li et al., 2021). The concatenation with the original input preserves raw coordinates alongside the sinusoidal basis.

A different line of work applies Fourier features not only at the input but throughout the depth of the network. In that setting, each pre-activation $z_l$ is mapped elementwise as

$h_l = [\sin(z_l), \cos(z_l)],$

so that every hidden layer emits a paired sine–cosine representation (Lewandowski et al., 2024). This “deep Fourier features” construction differs materially from shallow positional encodings because it changes the internal trainability and optimization geometry of the network, not merely its input parameterization.

Fourier features also appear in explicit kernel approximations. For stationary kernels, the feature map is designed so that

$k(x,x') \approx \langle z(x), z(x')\rangle,$

turning kernel learning into linear learning in feature space (Li, 2021, Wesel et al., 2021). In Gaussian process regression, the same idea yields low-rank covariance approximations with tractable training and prediction (Li et al., 2023).

2. Spectral viewpoint and kernel-theoretic basis

The theoretical basis for random Fourier features is the spectral representation of stationary kernels. For a stationary kernel $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$0, Bochner’s theorem permits the representation

$$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$1

so a finite-dimensional approximation can be formed by replacing the integral with sampled or deterministic frequencies (Wesel et al., 2021). In the standard random construction, frequencies are sampled from the spectral density and assembled into an explicit map such as

$$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$2

yielding a Monte Carlo approximation whose error decays at the usual $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$3 rate (Wesel et al., 2021).

This kernel perspective remains central even in neural-network applications. For coordinate MLPs, passing the input through a Fourier map changes the effective neural tangent kernel. The composed model behaves with a stationary kernel

$$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$4

so the inductive bias changes from a dot-product geometry to a shift-invariant one (Tancik et al., 2020). This is a central reason Fourier features alter what functions are learned quickly and which frequencies are favored during optimization.

In classification, random Fourier feature approximations inherit statistical guarantees from the underlying kernel method. For Lipschitz losses such as support vector machine and logistic regression, random Fourier feature classifiers can attain an $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$5 learning rate with only $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$6 features in the benign case $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$7, improving on earlier $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$8 feature requirements; under Massart’s low noise assumption, fast $$\gamma(v) = \left[ a_1 \cos(2\pi \mathbf b_1^\top v),\; a_1 \sin(2\pi \mathbf b_1^\top v),\; \ldots,\; a_m \cos(2\pi \mathbf b_m^\top v),\; a_m \sin(2\pi \mathbf b_m^\top v) \right],$$9 rates are also possible (Li, 2021). This places Fourier features at the intersection of kernel approximation and finite-sample statistical learning theory.

3. Spectral bias, frequency control, and optimization

A major reason Fourier features became prominent in deep learning is their relation to spectral bias. Standard coordinate-based MLPs preferentially learn low-frequency structure and are slow to fit high-frequency detail. In low-dimensional regression tasks, this limitation can be severe enough that plain MLPs fail to recover sharp spatial structure both in theory and in practice (Tancik et al., 2020). Fourier features mitigate this by furnishing the network with a basis already aligned with oscillatory components.

From the neural tangent kernel viewpoint, learning speed is governed by kernel eigenvalues. In one formulation, the training dynamics are approximated by

$\mathbf b_j$0

so components corresponding to small eigenvalues decay slowly (Tancik et al., 2020). Fourier embeddings alter the spectrum of the effective kernel and thereby accelerate fitting of higher-frequency modes.

In reinforcement learning, this spectral argument is sharpened into a functional regularization interpretation. For learned Fourier features analyzed in the infinite-width NTK regime, the residual under gradient flow evolves as

$\mathbf b_j$1

with learning rates determined by the eigenvalues of the NTK matrix $\mathbf b_j$2 (Li et al., 2021). For a two-layer learned-Fourier-feature model on the sphere, the kernel takes the form

$\mathbf b_j$3

equivalently

$\mathbf b_j$4

making the role of the Fourier initialization variance $\mathbf b_j$5 explicit (Li et al., 2021). Small $\mathbf b_j$6 yields broader, smoother kernels and suppresses high frequencies; large $\mathbf b_j$7 yields more localized kernels and faster learning of sharper structure. The paper interprets this as direct control over underfitting versus overfitting of frequencies, particularly relevant when Bellman targets are noisy (Li et al., 2021).

A related but distinct trainability argument appears in continual learning. Deep Fourier features, defined as concatenated sine and cosine activations in every layer, are motivated by the observation that for any pre-activation $\mathbf b_j$8, at least one of $\mathbf b_j$9 or $a_j$0 is locally well approximated by a linear function on an interval of width $a_j$1 (Lewandowski et al., 2024). This yields a dynamic balance between nonlinearity and an approximate linear pathway, which the authors connect to preservation of plasticity across non-stationary tasks (Lewandowski et al., 2024).

4. Variants and methodological families

Fourier features now comprise several distinct methodological families rather than a single technique.

Fixed random Fourier features

The classical random Fourier feature method samples frequencies independently from a spectral density and uses a finite sinusoidal map to approximate a stationary kernel (Wesel et al., 2021). Its chief advantages are simplicity, compatibility with linear solvers, and scalability relative to exact kernel methods. Its main limitation is Monte Carlo error and the resulting need for many features in difficult regimes (Wesel et al., 2021, Li, 2021).

Deterministic and quadrature-based features

To reduce stochastic approximation error, deterministic constructions replace Monte Carlo sampling by quadrature or structured spectral bases. Deterministic Fourier features can achieve exponentially decaying approximation error in the number of basis functions under suitable one-dimensional smoothness and domain assumptions, but naïve tensor-product extensions suffer from the curse of dimensionality (Wesel et al., 2021). Trigonometric Quadrature Fourier Features (TQFF) further adapt quadrature to oscillatory Fourier integrals by constructing a rule exact for trigonometric polynomials rather than ordinary polynomials, improving approximation accuracy and uncertainty calibration in Gaussian process regression with fewer features over a broad range of length-scales (Li et al., 2023).

Learned Fourier features

In learned Fourier features, the frequency matrix is optimized by backpropagation rather than fixed a priori (Li et al., 2021). This allows the embedding itself to adapt to the task, while retaining an interpretable frequency-control parameter through the initial variance of the basis. Learned bases are particularly relevant in high-dimensional reinforcement-learning settings where fixed random features can be inadequate (Li et al., 2021).

Deep Fourier features

Deep Fourier features replace conventional hidden activations with sine–cosine pairs at every layer (Lewandowski et al., 2024). This is not merely a richer basis for the input; it redefines the hidden state space and, according to the cited analysis, approximately embeds a deep linear network inside a nonlinear architecture. The aim is not primarily representation of high-frequency inputs, but sustained trainability in continual learning (Lewandowski et al., 2024).

Quantized and filtered Fourier embeddings

In implicit neural representations, Quantized Fourier Features (QFF) quantize Fourier-feature values into bins with learnable embeddings, combining periodic multiresolution structure with explicit local parameterization (Lee et al., 2022). A separate line robustifies Fourier embeddings by inserting a bias-free MLP filter acting multiplicatively on the Fourier channels,

$a_j$2

to suppress unnecessary frequencies or create sum-and-difference frequencies through channel interactions (Ma et al., 8 Feb 2025). This suggests a shift from treating Fourier features as fixed preprocessing to treating them as objects of adaptive frequency modulation.

Nonstationary and structured-domain extensions

Fourier features have also been generalized beyond stationary Euclidean kernels. Regular Fourier features for harmonizable Gaussian processes discretize the spectral representation directly on a regular frequency grid and preserve correlations among spectral weights, avoiding the probability-density interpretation used in stationary random Fourier features (Jawaid et al., 26 Feb 2026). Random Fourier Signature Features lift random Fourier approximations into the signature-kernel framework for sequences, producing unbiased explicit features for ordered, variable-length time series (Toth et al., 2023). These developments indicate that “Fourier features” increasingly denotes a general spectral-design paradigm rather than only Rahimi–Recht-style random features.

5. Major application domains

Coordinate-based neural representations

The modern revival of Fourier features is closely associated with low-dimensional coordinate regression in vision and graphics. Injecting sinusoidal embeddings into coordinate MLPs markedly improves image regression, 3D shape regression, CT reconstruction, MRI reconstruction, and NeRF-style inverse rendering (Tancik et al., 2020). The improvements are explained by the ability of the transformed model to learn high-frequency functions in low-dimensional domains that plain MLPs oversmooth (Tancik et al., 2020). Positional encodings used in neural radiance fields are a structured special case of this broader Fourier-feature framework.

Reinforcement learning

Learned Fourier features have been used to improve sample efficiency and stability in off-policy deep reinforcement learning (Li et al., 2021). In experiments on DeepMind Control Suite tasks with SAC, the architecture beats MLP baselines in 6 out of 8 state-based environments and yields clear gains on image-based benchmarks with SAC + RAD (Li et al., 2021). The reported gains concentrate in the critic rather than the policy, consistent with the hypothesis that frequency-space regularization is especially valuable where Bellman bootstrapping introduces noise (Li et al., 2021).

Continual learning

In continual supervised learning, deep Fourier features improve performance under label noise, class-incremental learning, and pixel permutations on CIFAR10, CIFAR100, and tiny-ImageNet by replacing ReLU activations with sine–cosine pairs (Lewandowski et al., 2024). The motivating claim is that linear function approximation and a special case of deep linear networks do not suffer from loss of plasticity, and deep Fourier features inherit part of this trainability while retaining nonlinear expressivity (Lewandowski et al., 2024).

Gaussian processes and scalable kernel methods

Fourier features remain central to scalable Gaussian process inference. Random, deterministic, quadrature, and RKHS-based Fourier features all reduce GP complexity by replacing dense kernel matrices with low-rank explicit feature maps (Li et al., 2023, Shi et al., 2024, Jawaid et al., 26 Feb 2026). In compositional and deep GP models, ODE-based RKHS Fourier response features modulate amplitude and phase through convolution with a Green’s function, extending fixed sinusoidal bases to dynamical settings (Shi et al., 2024).

Periodic and high-dimensional kernels

For multidimensional periodic kernels, Index Set Fourier Series Features approximate the kernel using deterministic multivariate Fourier series restricted to sparse index sets, outperforming random Fourier features on periodic regression problems and kernel reconstruction (Tompkins et al., 2018). This setting emphasizes a less publicized but historically central role of Fourier features: direct exploitation of harmonic structure in periodic covariance models.

Imitation learning and 3D perception

Point-cloud imitation learning has recently adopted axis-aligned NeRF-style Fourier features on Cartesian coordinates to improve high-precision robotic manipulation (Gyenes et al., 10 Jun 2026). The method maps each coordinate independently across multiple wavelengths and improves performance across several point-cloud encoders on RoboCasa, ManiSkill3, and real-robot tasks (Gyenes et al., 10 Jun 2026). The stated rationale is again spectral bias: raw Cartesian features are “slow-moving,” whereas Fourier mappings make fine geometric distinctions more directly available to the policy network (Gyenes et al., 10 Jun 2026).

Explainable AI and attribution

Fourier features have also become an object of attribution analysis. A recent method scores Fourier coefficients using an error-response construction

$a_j$3

arguing that deletion in Fourier space corresponds to genuine removal of signal components rather than introduction of pixel-space artifacts (Liu et al., 2 Apr 2025). In ImageNet experiments with Vision Transformers, only $a_j$4 of Fourier features are required to maintain the original predictions for $a_j$5 of samples (Liu et al., 2 Apr 2025). This suggests that Fourier features may function not only as engineered inputs but also as an explanatory basis for model behavior.

6. Theoretical interpretations and broader significance

Fourier features admit several complementary interpretations, none of which exhaust the concept.

First, they are explicit basis expansions that convert nonlinear function approximation into learning over a richer linear span of oscillatory atoms. In the simplest view, they expose high-frequency directions to a model that would otherwise discover them only slowly (Tancik et al., 2020).

Second, they are kernel approximators. Whether frequencies are random, quadrature-based, or learned, the feature map induces an approximate kernel, and much of the theory concerns approximation error, effective dimension, and generalization (Wesel et al., 2021, Li, 2021, Li et al., 2023).

Third, they are mechanisms for frequency-space regularization. The learned-Fourier-feature reinforcement-learning analysis makes this explicit by tying the initial basis variance to the underfitting–overfitting trade-off across frequencies (Li et al., 2021). This perspective differs from ordinary weight decay because it regulates the learned function rather than only parameter norms.

Fourth, they are tied to invariance and harmonic analysis more broadly. For networks invariant to a finite group, weights can be shown to recover the Fourier transform on that group, with matrix-valued irreducible unitary representations replacing scalar Fourier modes in the non-commutative case (Marchetti et al., 2023). This result places Fourier features within a larger algebraic theory of invariant representations rather than only the Euclidean sinusoidal setting.

A common misconception is that Fourier features are synonymous with positional encoding. Positional encodings are one important instance, especially in neural fields, but the literature also includes random and deterministic kernel approximations, learned frequency matrices, deep sine–cosine activations, response features shaped by ODE dynamics, and nonstationary spectral discretizations (Tancik et al., 2020, Li et al., 2021, Lewandowski et al., 2024, Shi et al., 2024, Jawaid et al., 26 Feb 2026).

Another misconception is that Fourier features invariably improve performance. The literature instead presents a recurring bias–variance or smoothness–detail trade-off. Too narrow a frequency spectrum induces underfitting and oversmoothing; too wide a spectrum can produce aliasing, overfitting, or noise sensitivity (Tancik et al., 2020, Li et al., 2021, Li et al., 2023, Ma et al., 8 Feb 2025). The choice of scale, distribution, or adaptive mechanism is therefore integral to the method rather than a peripheral hyperparameter.

7. Limitations, controversies, and open directions

Several limitations recur across the literature. Random Fourier features converge slowly by Monte Carlo sampling, motivating deterministic and adaptive alternatives (Wesel et al., 2021, Li et al., 2023). Deterministic tensor-product features can suffer from the curse of dimensionality unless coupled with low-rank tensor structure or sparse index sets (Wesel et al., 2021, Tompkins et al., 2018). In neural fields, fixed Fourier embeddings may introduce noisy outputs or unstable reconstructions when the sampled basis is mismatched to the target spectrum (Ma et al., 8 Feb 2025). In high-dimensional settings, fixed random features can be inferior to learned or structurally adapted bases (Li et al., 2021).

There is also no single answer to the question of whether Fourier features should be fixed or learned. One line of work reports that jointly training the Fourier frequencies in low-dimensional regression does not help much because the frequencies move little from initialization (Tancik et al., 2020). Another shows that training the basis matrix is important in high-dimensional reinforcement learning and that fixed random Fourier features work worse as dimensionality increases (Li et al., 2021). This suggests that the utility of learning the basis is strongly regime-dependent.

A further tension concerns robustness versus expressivity. Richer Fourier embeddings can recover fine detail, but they may also amplify high-frequency noise or degrade calibration. This tension motivates filtered embeddings in INRs (Ma et al., 8 Feb 2025), functional regularization in reinforcement learning (Li et al., 2021), and trigonometric quadrature constructions in Gaussian processes (Li et al., 2023). A plausible implication is that the future of Fourier features lies less in simply adding more frequencies than in designing task-aligned spectral priors and adaptive frequency controls.

Recent work points toward several extensions. Nonstationary Gaussian-process models now use regular-grid spectral discretizations that preserve cross-frequency correlations (Jawaid et al., 26 Feb 2026). Compositional GP models employ ODE-modulated RKHS Fourier response features with adaptive amplitude and phase (Shi et al., 2024). Attribution methods treat Fourier coefficients as the primary explanatory units for model decisions (Liu et al., 2 Apr 2025). Group-invariance theory links learned representations to generalized Fourier transforms on finite groups (Marchetti et al., 2023). Taken together, these developments suggest that Fourier features have evolved from a practical trick for positional encoding into a unifying spectral language spanning approximation theory, kernel methods, representation learning, reinforcement learning, continual learning, and explainable AI.