NeuralGrok: Frequency-Driven Grokking

• NeuralGrok is a phenomenon where networks initially memorize data, then abruptly generalize after aligning dominant low-frequency components.
• The framework uses frequency analysis to show that low-frequency modes are learned faster than high-frequency ones, leading to a coarse-to-fine spectral evolution.
• Experiments on synthetic regression, image recognition, and modular arithmetic validate that spectral sharpening and phase alignment in weights trigger generalization.

NeuralGrok designates a spectrum of mechanistic and experimental insights on the phenomenon of grokking in neural networks—where models, after rapidly reaching perfect training accuracy, exhibit a prolonged period of poor test performance before suddenly achieving excellent generalization. Recent research has established that NeuralGrok is fundamentally a frequency-structured, algorithmically latent, and phase-transition-driven process, which can be characterized and controlled by frequency analysis, function space dynamics, and explicit modeling of the evolution of learned features and spectral modes (Zhou et al., 2024, Swaroop, 24 Mar 2026).

1. Phenomenology: Definition and Empirical Manifestation

Grokking is defined as a two-phase dynamic in which a neural network, after an initial period of steeply declining training loss (with $\ell_{\mathrm{train}}(t)\to0$ for $t\in[0,T_1]$), remains highly overfit and exhibits stagnant or growing test loss ($\ell_{\mathrm{test}}(t)\gg0$ or increasing for $t\in[0,T_1]$). After many additional epochs, the test loss sharply transitions toward zero ($\ell_{\mathrm{test}}(t)$ drops for $t\in[T_1,T_2]$), marking the moment when the network "groks" the true rule underlying the data (Zhou et al., 2024). This abrupt memorization-to-generalization phenomenon is robust across real and synthetic datasets and a wide range of neural architectures.

2. Frequency-Based Functional Dynamics

A key rationale for NeuralGrok is rooted in the frequency perspective of function approximation. Any dataset $S = \{(x_i, y_i)\}_{i=1}^n$ and learned predictor $f(x;\theta)$ admit a nonuniform discrete Fourier transform (NUDFT) in any projection direction $d$: $$\mathcal F[S](k) = \frac{1}{n} \sum_{i=1}^n y_i e^{-ik x_i\cdot d}$$ The amplitude $t\in[0,T_1]$0 quantifies the salience of frequency $t\in[0,T_1]$1. Empirically, both real-world and synthetic datasets are dominated by a small number of low-frequency components, while small or nonuniform training samples introduce low-amplitude spurious peaks ("aliasing") (Zhou et al., 2024). Frequencies are defined as "salient" if $t\in[0,T_1]$2 and "less salient" otherwise.

The Frequency Principle (F-Principle) asserts that, under typical small initialization and gradient-based training, neural networks fit low-frequency targets more rapidly than high-frequency modes. In the NTK (infinite-width) regime under MSE loss, each frequency $t\in[0,T_1]$3 evolves as

$t\in[0,T_1]$4

with $t\in[0,T_1]$5 a decreasing function of $t\in[0,T_1]$6; thus, lower frequencies ($t\in[0,T_1]$7 small) are learned fastest (Zhou et al., 2024). This results in a coarse-to-fine, spectrum-driven learning trajectory.

3. Spectral and Algorithmic Structure Preceding Generalization

Mechanistic studies of modular arithmetic tasks reveal that, even during the memorization phase (with chance-level generalization), the weights of trained ReLU MLPs encode latent algorithmic structure in Fourier space (Swaroop, 24 Mar 2026). For each hidden neuron $t\in[0,T_1]$8 with input weights $t\in[0,T_1]$9 (for modulus $\ell_{\mathrm{test}}(t)\gg0$0), one computes the DFT: $\ell_{\mathrm{test}}(t)\gg0$1 The dominant frequency $\ell_{\mathrm{test}}(t)\gg0$2 and phase $\ell_{\mathrm{test}}(t)\gg0$3 are extracted from $\ell_{\mathrm{test}}(t)\gg0$4, while output layer weights $\ell_{\mathrm{test}}(t)\gg0$5 exhibit the same $\ell_{\mathrm{test}}(t)\gg0$6 and a phase $\ell_{\mathrm{test}}(t)\gg0$7 satisfying $\ell_{\mathrm{test}}(t)\gg0$8.

Prior to grokking, input weights resemble near-binary square waves: $\ell_{\mathrm{test}}(t)\gg0$9 Over training, weight decay and gradient flow drive these weights to sharper, more perfectly binarized patterns, aligning the phase-sum relation and amplifying constructive interference at the correct modular output.

Even weights extracted from models that never grok (due to label noise) can be assembled into an "idealized" MLP which achieves high test accuracy. Thus, grokking is not the discovery of a novel circuit, but the refinement and binarization of a circuit already present in the memorizing model's weights (Swaroop, 24 Mar 2026).

4. Generalization Triggered by Frequency Alignment and Algorithmic Consolidation

NeuralGrok, in this mechanistic picture, is the sharpening and spectral binarization of latent algorithmic features until the network's frequency-phase structure is sufficiently aligned for global generalization. At the moment of grokking, the internal representation transitions from capturing idiosyncratic, less-salient (spurious) frequencies to a minimal, precise algorithm characterized by the correct, high-salience frequencies and their phase relationships.

This alignment of algorithmic structure and its associated reduction in weight and activation entropy are central to explaining the transition to generalization, as the test-loss plateau collapses when the spectral and algorithmic redundancy across the model reaches a critical threshold. The progression is quantitatively captured by tracking the amplitude evolution of each frequency mode and the emergence of sharpened, phase-sum-satisfying circuits in weight space (Swaroop, 24 Mar 2026).

5. Experimental Support and Validation

Zhou et al. validate the frequency-based NeuralGrok framework across three representative tasks (Zhou et al., 2024):

• One-dimensional synthetic regression: For $t\in[0,T_1]$0 on $t\in[0,T_1]$1, with various sampling schemes (uniform and nonuniform), nonuniform training introduces ambiguity in the learning of low-salience frequencies, resulting in delayed test generalization until spectral alignment is achieved.
• Real-world datasets (e.g., MNIST): The few dominant Fourier components correspond to genuine patterns in the data, with grokking appearing when training initially fits less salient (often spurious) test frequencies, then shifts to the salient spectrum.
• Fourier tracking: NUDFT or DFT analysis quantifies the spectral evolution of both model predictions and label targets, directly demonstrating the coarse-to-fine cascade and abrupt spectral collapse at grokking.

In modular addition, DFT-based diagnostics reveal phase-sum structure across the model before generalization and confirm the sharpening/binarization of input square waves during the grokking transition (Swaroop, 24 Mar 2026).

6. Implications and Theoretical Consequences

NeuralGrok provides a unifying lens for understanding delayed generalization phenomena in neural networks. The interplay between initial frequency fitting, progressive spectral sharpening, and algorithmic alignment elucidates why overparameterized models may require many more epochs to generalize than to memorize. This paradigm reframes grokking as a rigorously analyzable dynamic in function space, driven by spectral learning rates and phase relations, not architectural peculiarities or explicit algorithmic search.

The frequency perspective suggests practical diagnostics for anticipated grokking—such as tracking Fourier-mode amplitudes or the phase relationships in trained weights—and indicates directions for accelerating generalization through regularization or architectural alignment with the dominant frequencies in the data (Zhou et al., 2024, Swaroop, 24 Mar 2026). These findings establish NeuralGrok as the canonical frequency-phase framework for analysing and manipulating the mechanics of grokking in modern deep learning.