Frequency Principle in Deep Learning

• Frequency Principle is a training dynamic where neural networks first learn coarse, low-frequency features (smooth structures) before fitting fine, high-frequency details.
• Empirical studies using Fourier diagnostics, NUDFT, and Gaussian filtering show that low-frequency errors decay faster than high-frequency errors across diverse datasets and optimizers.
• This spectral bias underpins implicit regularization, influencing training dynamics, generalization performance, and the design of network architectures.

Searching arXiv for the specified paper and closely related Frequency Principle work to ground the article in current literature. Frequency Principle (F-Principle, FP), also known as spectral bias, is the empirical and theoretical tendency that, during training, deep neural networks (DNNs) tend to fit target functions from low to high frequency: low-frequency components are learned first, and higher-frequency components are learned later (Xu et al., 2019, Xu et al., 2022). In this context, “frequency” refers to the spectral content of the learned input-output mapping rather than the spatial frequency of raw inputs; low frequencies correspond to smooth, slowly varying structure, whereas high frequencies correspond to rapidly varying or oscillatory structure (Xu et al., 2019). Across synthetic regression, image classification, kernel-regime analyses, and several optimizer classes, FP has become a central description of training dynamics, implicit regularization, and the relative ease of learning smooth versus oscillatory targets (Ma et al., 2021, Luo et al., 2019).

1. Definition and spectral observables

FP is usually stated as an ordering property of training dynamics: the error associated with lower frequencies decays earlier or faster than the error associated with higher frequencies (Xu et al., 2019, Luo et al., 2019). For one-dimensional targets and predictions, the basic diagnostic is Fourier-domain relative error. In the notation used for sampled data,

$$\hat f_k = \frac{1}{n}\sum_{j=0}^{n-1} f(x_j)\, e^{-i 2\pi jk/n}, \qquad \hat f_{\theta,k}(t) = \frac{1}{n}\sum_{j=0}^{n-1} f_{\theta}(x_j,t)\, e^{-i 2\pi jk/n},$$

and the tracked quantity is

$$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$

FP is deemed to hold if $\Delta_F(k,t)$ approaches $0$ in order of increasing $k$, namely low $k$ first and higher $k$ later (Ma et al., 2021).

For high-dimensional data, explicit multidimensional Fourier analysis is often computationally prohibitive. Two proxy constructions recur in the literature. The first is the nonuniform discrete Fourier transform (NUDFT) along a selected direction, such as the first principal component, which measures the frequency content of the response mapping on irregularly sampled data (Xu et al., 2019). The second is Gaussian filtering, which defines low- and high-frequency components on the sample set: $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$ with $$y_i^{\text{high},\delta} = y_i - y_i^{\text{low},\delta}$$, and similarly for the network outputs $h_i=f_\theta(x_i)$ (Ma et al., 2021, Xu et al., 2019). The corresponding bandwise errors are

$$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$0

In this formulation, FP holds if $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$1 across training for multiple bandwidths $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$2 (Ma et al., 2021).

A second terminological point is important. In the original high-dimensional studies, “frequency” is the frequency content of the mapping $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$3, not the spatial frequency of the raw images themselves (Xu et al., 2019). This distinction is essential in classification settings, where Fourier diagnostics are applied to labels and network outputs as functions over input space.

2. Empirical evidence and universality across optimizers

The classical empirical picture was established on both synthetic and real datasets. On the one-dimensional target

$$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$4

with $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$5 evenly spaced points in $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$6, a tanh network learns the lowest-frequency peak first, then the middle peak, then the highest (Xu et al., 2019). In higher-dimensional tasks, NUDFT and Gaussian-filter diagnostics on MNIST, CIFAR-10, and VGG16 consistently show that low-frequency errors decrease rapidly and become small early in training, while high-frequency errors decrease much later (Xu et al., 2019).

A major extension is that FP is not contingent on gradient descent per se. The paper “Frequency Principle in Deep Learning Beyond Gradient-descent-based Training” shows that FP exists stably in DNN training under non-gradient-descent-based optimizers, including methods that use gradient information and methods that do not (Ma et al., 2021). The tested optimizers include Conjugate Gradient (CG), Truncated Newton (TNC/Newton-CG), BFGS, L-BFGS, Powell’s method, Particle Swarm Optimization (PSO), and a Monte-Carlo-like local random search (Ma et al., 2021).

The reported setups are deliberately varied. For one-dimensional synthetic regression, the targets are $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$7 and $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$8, trained with fully connected sigmoid networks such as $$\Delta_F(k,t) = \frac{|\hat f_{\theta,k}(t) - \hat f_k|}{|\hat f_k|}.$$9-$\Delta_F(k,t)$0-$\Delta_F(k,t)$1-$\Delta_F(k,t)$2, $\Delta_F(k,t)$3-$\Delta_F(k,t)$4-$\Delta_F(k,t)$5-$\Delta_F(k,t)$6, $\Delta_F(k,t)$7-$\Delta_F(k,t)$8-$\Delta_F(k,t)$9, and $0$0-$0$1-$0$2-$0$3 under MSE loss (Ma et al., 2021). On MNIST, only CG and L-BFGS were used because of memory constraints, with a sigmoid CNN followed by fully connected layers, trained on a subset of $0$4 randomly selected images with one-hot labels (Ma et al., 2021). For MNIST, Gaussian filtering with $0$5 and $0$6 gives bandwise errors satisfying $0$7 through training, indicating faster convergence of low-frequency content in both labels and predictions (Ma et al., 2021).

Optimizer-specific behavior differs in efficiency, but not in the basic spectral ordering. CG on the three-peak target shows early decay at $0$8 and later decay at $0$9; BFGS and L-BFGS preserve the same progression; Powell’s method, despite being derivative-free and slow on larger DNNs, still learns low-frequency components first; PSO and the Monte-Carlo-like scheme likewise show early reduction of low-frequency error and delayed fitting of higher-frequency components (Ma et al., 2021). TNC can fit low-frequency components first but may fit high frequencies imperfectly because search directions can fail to be descent when the Hessian is indefinite; nevertheless, the low-to-high order remains visible (Ma et al., 2021). This broad optimizer invariance is one of the strongest empirical arguments that FP is primarily a property of model class and induced dynamics in function space, rather than a peculiarity of plain gradient descent.

3. Mechanistic theories: activations, kernels, and explicit frequency dynamics

One influential explanation traces FP to the regularity of common activation functions. For a one-hidden-layer tanh network,

$k$0

the Fourier transform of the activation is

$k$1

and for an affine argument,

$k$2

Because this decays as $k$3 for large $k$4, the initial spectrum and the frequency-wise gradients are low-frequency dominated when weights are small (Xu et al., 2019). In that setting, Theorem 1 and Theorem 2 show that, near initialization, gradients from lower frequencies dominate those from higher frequencies and the low-frequency loss decreases faster with high probability (Xu et al., 2019).

The paper “Theory of the Frequency Principle for General Deep Neural Networks” generalizes this picture to multilayer networks, general activation functions, population densities, and a large class of loss functions (Luo et al., 2019). Its analysis separates initial, intermediate, and final stages of training and proves bounds of the form

$k$5

or

$k$6

showing that high-frequency contributions to training change are polynomially suppressed relative to low-frequency contributions (Luo et al., 2019). The mechanism is Sobolev regularity: smoothness of the network mapping and its parameter derivatives induces decay of high-frequency Fourier components.

A more explicit formulation appears in the linear frequency-principle (LFP) program for infinite-width two-layer networks in the NTK regime. For two-layer ReLU networks, the residual $k$7 obeys an exact or effective frequency-domain dynamics of the form

$k$8

with

$k$9

where $k$0 and $k$1 are determined by initialization statistics (Zhang et al., 2019). This directly makes lower frequencies evolve faster. In the exact NTK-based derivation for general activations, high frequencies evolve polynomially slower for ReLU-like nonsmooth activations and exponentially slower for smooth activations such as tanh (Luo et al., 2020). These works also show that the long-time limit of the linearized dynamics is equivalent to minimizing an explicit FP-norm, in which higher frequencies are more heavily penalized (Zhang et al., 2019, Luo et al., 2020). The corresponding a priori generalization bounds scale with the FP-norm of the target, making the link between spectral bias and generalization mathematically explicit (Zhang et al., 2019, Luo et al., 2020).

Kernel perspectives provide a parallel explanation. In wide-network regimes, training dynamics linearize around initialization and the evolution of Fourier modes can be written schematically as

$k$2

where $k$3 typically decreases with $k$4 for common architectures and activations (Ma et al., 2021). This yields larger effective learning rates for low frequencies and slower evolution of high frequencies.

4. Depth and the Deep Frequency Principle

FP concerns the spectral order in the learned function itself. A depth-specific refinement, the Deep Frequency Principle (DFP), concerns the effective target function seen by deeper layers (Xu et al., 2022, Xu et al., 2020). The construction separates a feedforward network into a pre-condition component and a learning component. With recursive notation

$k$5

the effective training set for the learning component at depth $k$6 is

$k$7

The empirical claim is: the effective target function for a deeper hidden layer biases towards lower frequency during the training (Xu et al., 2022, Xu et al., 2020).

Because direct high-dimensional Fourier transforms are impractical, DFP is diagnosed using Gaussian low-pass filtering and two derived quantities: the Low Frequency Ratio (LFR) and the Ratio Density Function (RDF) (Xu et al., 2022). For filtered labels,

$k$8

the LFR at effective cutoff $k$9 is

$k$0

and

$k$1

approximated numerically by finite differences (Xu et al., 2022). Leftward shifts of the RDF toward smaller $k$2 indicate increasing low-frequency bias.

Empirically, this effect was shown in ResNet-18 variants on CIFAR-10 and in fully connected tanh networks on MNIST (Xu et al., 2022, Xu et al., 2020). In the CIFAR-10 experiments, deeper variants reached fixed accuracy in fewer epochs and generalized better after $k$3 epochs; RDFs of the effective target function for the last hidden layer moved toward lower frequency content more quickly in deeper networks (Xu et al., 2022). In the MNIST fully connected network $k$4-$k$5-$k$6-$k$7-$k$8-$k$9-$$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$0, RDFs across hidden layers showed that deeper hidden layers saw increasingly low-frequency effective targets earlier in training (Xu et al., 2022). Very deep fully connected networks without residual connections could violate DFP because of vanishing gradients, while residual connections mitigated this and produced near-saturation of the depth effect (Xu et al., 2022, Xu et al., 2020).

The significance of DFP is interpretive rather than merely descriptive. Because low-frequency targets are learned faster by standard deep networks, deeper pre-conditioning can make the learning component face an easier, spectrally smoother problem (Xu et al., 2022). This provides an empirical explanation for the observation that deeper learning can be faster.

5. Generalization, numerical analysis, and design implications

FP is closely tied to implicit regularization and generalization. Real datasets such as MNIST and CIFAR-10 show low-frequency dominance in the Fourier domain of the response mapping, and the trained network output aligns with these low-frequency components on both training and test sets (Xu et al., 2019). This supports the interpretation that DNNs tend to fit training data by a low-frequency function, which helps explain good generalization on most real datasets (Xu et al., 2019). By contrast, parity functions and randomized datasets are high-frequency dominated; with limited samples, aliasing-induced artificial low-frequency components are fitted first, while the true high-frequency structure is not recovered, explaining poor generalization (Xu et al., 2019).

The same spectral viewpoint yields a sharp contrast with classical iterative numerical schemes. For the Jacobi method applied to discretized one-dimensional Poisson equations, the error evolution has eigenvalues

$$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$1

so high-frequency modes decay faster and low-frequency modes converge more slowly (Xu et al., 2019). DNN training exhibits the opposite ordering: low-frequency peaks converge first, high-frequency peaks later (Xu et al., 2019). This complementarity motivates hybrid PDE solvers in which a DNN first captures low frequencies and a classical method then removes the remaining high-frequency residuals (Xu et al., 2019).

From an engineering standpoint, a recurrent conclusion is that changing the optimizer is not a reliable way to remove slow learning of high frequencies. Because FP appears across gradient-based and gradient-free optimizers, optimizer choice should be made for stability and efficiency, with the expectation that spectral bias will persist (Ma et al., 2021). When high-frequency fidelity matters, the proposed remedies target representation and architecture instead: Fourier features, positional encodings, multiscale or skip-connection designs such as U-Nets, multi-resolution CNNs, and MscaleDNN, activations or initializations with richer high-frequency spectra, and curriculum or residual-based training schemes that emphasize higher-frequency content later in optimization (Ma et al., 2021, Xu et al., 2022, Xu et al., 2019).

Several practical diagnostics follow directly from FP. Band-wise errors can be monitored during training; early stopping can preserve low-frequency structure while avoiding later fitting of noise; and frequency-aware regularization or loss weighting can be used when particular bands are critical (Ma et al., 2021). For high-dimensional frequency diagnostics, Gaussian filtering remains the standard tractable proxy, though it measures coarse low/high decompositions rather than a full multidimensional Fourier spectrum (Xu et al., 2019, Ma et al., 2021).

6. Exceptions, domain dependence, and newer extensions

FP is widely observed, but it is not best interpreted as an inviolable rule. On the curved domain $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$2, “frequency” is indexed by spherical harmonic degree $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$3 rather than Euclidean $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$4 (Zhai, 24 Aug 2025). For shallow ReLU networks of the form

$$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$5

the single-neuron feature $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$6 has spherical harmonic coefficients $$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$7 with asymptotic decay

$$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$8

and in the trainable-direction case the additional rotation term contributes at order

$$y_i^{\text{low},\delta} = \frac{1}{C_i} \sum_{j=0}^{n-1} y_j\, G^{\delta}(x_i - x_j), \qquad G^{\delta}(x) = \exp\!\big(-\|x\|^2/(2\delta)\big),$$9

(Zhai, 24 Aug 2025). These formulas imply a strong low-frequency tendency on the sphere, but the same paper gives precise conditions for violation: degenerate equilibria at initialization, incorrect update signs, vanishing error at representative points, high-frequency-initialized outputs such as $$y_i^{\text{high},\delta} = y_i - y_i^{\text{low},\delta}$$0, or highly non-uniform direction sets can produce partial or strong violations (Zhai, 24 Aug 2025). The paper therefore argues that, on $$y_i^{\text{high},\delta} = y_i - y_i^{\text{low},\delta}$$1, FP should be viewed as a tendency rather than a rule (Zhai, 24 Aug 2025).

This nuanced view is consistent with more general caveats already present in the earlier literature. Most direct non-gradient-descent evidence comes from one-dimensional sinusoidal regression; in high-dimensional vision settings, only CG and L-BFGS were feasible on MNIST because of memory and computation limits (Ma et al., 2021). The tested architectures in that study were fully connected sigmoid networks and a small sigmoid CNN; other activations and deeper architectures may alter rates even if the low-to-high order persists (Ma et al., 2021). The Gaussian filter used in high dimensions is an approximation to low/high band decomposition rather than a true multidimensional Fourier split (Ma et al., 2021).

Recent quantum machine learning papers extend the FP vocabulary beyond classical DNNs, but not always in the same way. One line formulates a task-adaptive FP for parameterized quantum circuits (PQCs): during gradient-based training, the model preferentially learns frequencies within the primary frequency range of the objective function, whether those dominant frequencies are low, medium, or high (Xu et al., 2024). Another line gives a unified manifold-based theorem for classical and quantum models under gradient flow, in which low-frequency loss decays faster than high-frequency loss and aligned Pauli noise suppresses the Fourier component labeled by $$y_i^{\text{high},\delta} = y_i - y_i^{\text{low},\delta}$$2 by the factor

$$y_i^{\text{high},\delta} = y_i - y_i^{\text{low},\delta}$$3

thereby exponentially attenuating high-frequency terms (Lu et al., 6 Jan 2026). This suggests that, outside the classical Euclidean DNN setting, “frequency principle” can denote related but not identical spectral orderings, depending on the representation, geometry, and admissible frequency set.

Taken together, the literature supports a stable core claim: under standard training, neural models usually learn coarse structure before fine detail. What varies across domains is the exact definition of frequency, the analytic object that controls convergence rates, and the conditions under which the low-to-high ordering may weaken, saturate, or fail (Luo et al., 2019, Zhai, 24 Aug 2025).