Neural Network Grokking Dynamics

Updated 5 July 2026

Neural network grokking is defined as a delayed generalization regime featuring early memorization followed by a sudden shift to strong test performance.
It illustrates how interpolation and feature learning decouple, transitioning from initial lazy kernel-like fitting to task-aligned representations.
Grokking is influenced by data fractions, regularization, and optimization dynamics, offering insights into late-stage learning and model robustness.

Neural network grokking denotes a delayed-generalization regime in which a model reaches near-perfect or perfect training performance while test performance remains poor or stagnant, and only much later undergoes an abrupt transition to strong generalization. First isolated on small algorithmic datasets such as modular arithmetic and finite-group operation tables, the phenomenon is now studied in MLPs, transformers, CNNs, reinforcement learning agents, tensor-network models, and scientific classification tasks. Across these settings, grokking is used as a probe of how interpolation, representation formation, robustness, and late-stage optimization dynamics can decouple in overparameterized systems (Power et al., 2022, Kumar et al., 2023, Sakamoto et al., 25 Sep 2025).

1. Canonical phenomenology and experimental regimes

The canonical profile of grokking consists of three phases: early memorization, a long plateau, and a late generalization jump. In the original algorithmic setting, small transformers trained on finite binary-operation tables often attain near-perfect training accuracy within $10^3$ – $10^4$ updates, while validation accuracy remains near chance for orders of magnitude more steps, before rising sharply toward $99\%$ – $100\%$ . A standard example is modular division mod $97$ with a $50\%$ train split, where training accuracy becomes near-perfect in fewer than $10^3$ steps, validation accuracy stays at chance until around $10^5$ steps, and only approaches perfect generalization near $10^6$ steps (Power et al., 2022).

This basic pattern is sensitive to data fraction. On algorithmic tables, smaller training fractions greatly increase time-to-grok, while sufficiently large fractions make train and test curves move together. In a simplified transformer for modular addition mod $97$, one reported regime uses $10^4$ 0 training data: training accuracy reaches $10^4$ 1 at about $10^4$ 2 steps, validation improves sharply around $10^4$ 3 steps, and reaches $10^4$ 4 by about $10^4$ 5 steps; at $10^4$ 6, no generalization appears even after $10^4$ 7 steps, while at $10^4$ 8 validation rises almost as quickly as training accuracy (Qiye et al., 2024).

Although the phenomenon was first associated with small algorithmic datasets, later work broadened its scope. Delayed generalization or closely related late-phase transitions have been reported in MNIST MLPs, CIFAR-10 warm-start protocols, CNNs on CIFAR-10 and CIFAR-100, ResNet18 on Imagenette, Atari agents, tensor-network classifiers, and even MLPs trained on 2D Ising configurations. Some studies also distinguish classical grokking from adjacent regimes such as delayed robustness, where adversarial accuracy improves well after interpolation, or abrupt learning reorganizations on tasks without a held-out split, such as XOR, where the transition is interpreted as a change in gradient geometry rather than classical delayed test generalization (Humayun et al., 2024, Lyle et al., 26 Jul 2025, Wang, 6 Apr 2026, Hutchison et al., 29 Oct 2025).

2. Feature learning, representation change, and mechanistic accounts

A major line of work explains grokking as a transition from lazy training dynamics to rich feature learning. In this view, training initially behaves like kernel regression around initialization,

$10^4$ 9

so the network fits the training set largely with fixed initial features. If the target function is poorly aligned with the top eigendirections of the initial NTK, this early solution interpolates training data but generalizes poorly. Only later, when the network leaves the linearized regime and its internal features change substantially, does it discover a task-aligned representation, producing the late test-loss drop (Kumar et al., 2023).

A closely related account ties grokking to the scale of movement in parameter space relative to parameter norm. The proposed control variable is the movement ratio

$99\%$ 0

with $99\%$ 1. For scale-invariant objectives, the effective learning rate decreases as parameter norms grow, pushing optimization toward a lazy regime; increasing $99\%$ 2 or controlling norms can re-enter a rich regime with substantial representational change. This framework motivates “ELR re-warming,” often paired with Normalize-and-Project (NaP), and is used to connect grokking to primacy bias in nonstationary learning, warm-start image classification, and reinforcement learning (Lyle et al., 26 Jul 2025).

Several studies sharpen the representational picture. On modular arithmetic, fully connected two-layer networks with quadratic activation admit analytic solutions in terms of periodic feature maps on $99\%$ 3: the network learns cosine-like or character-like features whose constructive interference realizes the target modular constraint. Empirically, gradient descent and AdamW recover the same periodic structure, making late generalization interpretable as the emergence and phase-alignment of task-specific Fourier features (Gromov, 2023). A frequency-based account reaches a related conclusion from a different angle: nonuniform or insufficient sampling can create spurious low-frequency components in the training spectrum, and early training fits those components first; only later does the model align with the salient frequencies of the full or test distribution, at which point generalization appears (Zhou et al., 2024).

Another mechanistic proposal places the embedding layer at the center of grokking. In modular arithmetic MLPs, explicit embeddings induce delayed generalization, whereas MLPs without embeddings can generalize immediately on modular addition. Two causes are emphasized: sparse, frequency-skewed embedding updates under weight decay, which slow learning for rare tokens, and bilinear coupling between embeddings and downstream weights, which introduces saddle structure and asymmetric curvature. The resulting recommendation is an adaptive learning-rate ratio,

$99\%$ 4

to rebalance embedding and downstream dynamics (AlquBoj et al., 21 May 2025).

A further geometric account studies local complexity through the density of piecewise-linear regions. Here grokking is associated with a late migration of linear-region boundaries away from training samples and toward decision boundaries. The network first uses many local regions to interpolate, then later smooths the map around data while concentrating complexity where classification margins matter, thereby enabling both delayed generalization and delayed robustness (Humayun et al., 2024).

3. Regularization, optimization, and distributional explanations

Regularization is one of the oldest and most persistent empirical levers associated with grokking. In the original modular-arithmetic experiments, explicit $99\%$ 5 weight decay in AdamW markedly improved data efficiency, more than halving the training fraction required for high validation accuracy within a fixed $99\%$ 6-step budget relative to most ablations. Minibatch noise and explicit Gaussian noise also improved generalization relative to full-batch optimization, while learning rate had to lie within a fairly narrow window for grokking to occur reliably (Power et al., 2022).

One formal robustness account makes this intuition precise under explicit assumptions. For MSE-trained interpolating networks with Lipschitz input Jacobian, the paper derives a robustness radius $99\%$ 7 that increases as $99\%$ 8 and sharpness decrease, and proves that if a $99\%$ 9-fraction of test points lies within that radius of training examples, test accuracy is at least $100\%$ 0. In this sense, the $100\%$ 1 weight norm provides a sufficient condition for grokking, while perturbation-based training and a commutativity regularizer accelerate generalization on modulo addition by inducing robustness and a necessary algebraic invariant (Tan et al., 2023).

Other work broadens the role of regularization beyond Euclidean norm. One analysis shows that if there exists an interpolating solution with a property $100\%$ 2 that generalizes, such as sparsity or low rank, then gradient descent with a small nonzero regularizer toward $100\%$ 3, for example $100\%$ 4 or nuclear norm, produces delayed generalization with a delay scaling like $100\%$ 5, where $100\%$ 6 is learning rate and $100\%$ 7 the regularization strength. The same study argues that $100\%$ 8 norm is not a reliable universal proxy when the relevant inductive bias is not Euclidean, and that added depth can induce or suppress grokking through implicit regularization even without explicit penalties (Notsawo et al., 6 Jun 2025).

A separate statistical explanation shifts attention from optimization to train–test mismatch. In that account, small data are associated with grokking not because sparsity itself is causal, but because small or imbalanced samples conveniently induce subclass-level distribution shift between $100\%$ 9 and $97$0. On synthetic datasets with controlled subclass under-sampling, removing an entire subclass can prevent grokking in equidistant settings, while reintroducing even a small fraction can recover it; in equivariant settings, delayed generalization can appear even with zero samples from a subclass because related subclasses transfer structure. The same paper reports grokking-like behavior on dense datasets and under minimal hyperparameter tuning, provided the train–test shift is maintained (Carvalho et al., 3 Feb 2025).

A later geometric framework links these late effects to neural collapse. It argues that the decisive quantity is the contraction of population within-class variance in representation space. Training loss can plateau quickly, but the collapse of within-class variance—and hence population-level class concentration—proceeds on a slower time scale, especially when weight decay is small. This yields a direct route from late neural-collapse dynamics to both delayed generalization and information-bottleneck-style compression (Sakamoto et al., 25 Sep 2025).

4. Phase-transition and statistical-mechanical interpretations

The grokking literature contains several explicit phase-transition analogies, but they are not all the same. One analytical approach on two-layer teacher–student models formulates grokking as a first-order phase transition in an adaptive-kernel theory of feature learning. In that framework, the posterior over input weights develops teacher-aligned saddles, and the post-grokking state is a mixed phase in which specialized internal representations coexist with pre-transition ones. The resulting picture is discontinuous and beyond static NTK or GP descriptions (Rubin et al., 2023).

A different proposal studies gradient avalanche dynamics and introduces an effective dimensionality $97$1 through finite-size scaling,

$97$2

Across eight model scales, the aggregate exponent is $97$3, but phase-split analysis yields $97$4, $97$5, and $97$6. In this interpretation, grokking is a dimensional phase transition from sub-diffusive to super-diffusive gradient geometry, with self-organized-criticality-like avalanche statistics localized around the transition (Wang, 6 Apr 2026).

A statistical-mechanical alternative rejects a simple barrier-crossing first-order story. By sampling the joint entropy landscape $97$7 with Wang–Landau molecular dynamics on modular arithmetic transformers, one study reports no entropy barrier between memorization and generalization at comparable training loss. It instead interprets grokking as a computational glass relaxation: training rapidly quenches the model into a non-equilibrium memorizing state, and delayed generalization is a slow relaxation toward higher-entropy, better-generalizing configurations at nearly the same energy. The associated WanD optimizer eliminates the plateau on modular addition and finds high-norm generalizing solutions with $97$8, challenging explanations that rely exclusively on weight norms entering a narrow “Goldilocks zone” (Zhang et al., 16 May 2025).

Yet another account returns to activated escape, but in a more structured setting. In deep linear networks with $97$9-induced first-order phase transitions, metastable low-rank phases are separated by finite barriers, and SGD noise acts as an effective temperature $50\%$ 0. The measured escape times obey Arrhenius scaling,

$50\%$ 1

with $50\%$ 2 and $50\%$ 3 in the reported experiments. In this view, grokking is noise-driven escape from metastable phases, and sparse sub-sampling reproduces the canonical delayed closure of the train–test gap (Ersoy et al., 15 Jun 2026).

Quantum-inspired and scientific-task studies offer parallel transition pictures. In MPS classifiers, grokking coincides with an entanglement transition from volume-law to sub-volume-law behavior, vanishing label-space coherence, and “eigenvalue evaporation” in the entanglement spectrum (Pomarico et al., 13 Mar 2025). In an Ising-energy classifier, delayed generalization is accompanied by a transformation from a dense network to a relatively sparse subnetwork, with a dip in the output confidence gap and a collapse of PCA-based gradient complexity measures during the transition (Hutchison et al., 29 Oct 2025). These formulations suggest that the phrase “phase transition” in the grokking literature names a family of related but non-identical reorganizations in feature space, gradient space, entropy landscapes, or effective subnetworks.

5. Diagnostics and measurable precursors

Because grokking is primarily a dynamical phenomenon, much of the literature focuses on observables that change before or during the late transition. In lazy-to-rich analyses, the central diagnostics are NTK drift, task–kernel alignment, and sufficient statistics of the evolving representation. One such statistic is centered-kernel alignment,

$50\%$ 4

which measures alignment between labels and the initial kernel, while another is direct monitoring of NTK drift $50\%$ 5 (Kumar et al., 2023).

Feature-learning accounts introduce explicit probes of representational motion. For a layer $50\%$ 6, the normalized feature covariance

$50\%$ 7

and its change

$50\%$ 8

track rotation of learned features. ReLU activation-pattern change,

$50\%$ 9

plays a similar role, while attention output rank and attention-mask entropy are useful in transformer settings. Rising $10^3$ 0 and $10^3$ 1 indicate entry into a rich feature-learning regime and are central to ELR-based re-warming methods (Lyle et al., 26 Jul 2025).

Robustness-based diagnostics emphasize input sensitivity rather than feature geometry. Perturbation error, sharpness $10^3$ 2, and the matrix-information-theoretic quantities PMI and PE, computed on perturbed inputs, are reported to align tightly with the grokking moment, whereas raw $10^3$ 3 norm often begins changing earlier and is not temporally precise. Their differences relative to the unperturbed case, MID and ED, are proposed as early indicators of eventual grokking (Tan et al., 2023).

Higher-order information measures supply another precursor family. Using O-information

$10^3$ 4

one study interprets negative $10^3$ 5 as synergy-dominated computation and finds an early synergy peak before grokking in modular addition. The cardinality of the subset attaining maximal synergy is treated as the size of a “synergistic sub-network,” and isolating that sub-network can reproduce or accelerate the original model’s generalization dynamics (Clauw et al., 2024).

Several newer works focus on practical forecast metrics. One introduces the Dropout Robustness Curve $10^3$ 6, measuring test accuracy at inference-time dropout rate $10^3$ 7, and reports that post-grokking models remain accurate for small $10^3$ 8 and degrade only at moderate dropout. The same study finds a local maximum in the variance of test accuracy under Monte Carlo dropout near grokking, a decrease in the percentage of inactive ReLU neurons, and a convergence of embeddings toward a bimodal distribution with symmetric peaks near $10^3$ 9 on modular addition mod $10^5$ 0 (Salah et al., 15 Jul 2025). At a still broader level, the neural-collapse view advocates RNC1, NC1, and NC2 as late-phase indicators because within-class variance contraction is directly tied to both test-error bounds and information compression (Sakamoto et al., 25 Sep 2025). The dimensional-phase-transition account similarly proposes monitoring the crossing of $10^5$ 1 through the random-diffusion baseline $10^5$ 2 as a trainability diagnostic (Wang, 6 Apr 2026).

6. Interventions, applications, and open problems

Because grokking exposes a long plateau between interpolation and generalization, many papers ask how to induce, accelerate, or suppress it. One of the most general proposals is ELR re-warming: keep parameter norms under control with NaP, then cyclically or adaptively raise the learning rate to transiently increase $10^5$ 3 and induce feature learning when generalization stalls. This method is reported to accelerate modular-arithmetic grokking, close warm-start generalization gaps on CIFAR-10, and mitigate primacy bias in continual learning and Atari reinforcement learning, especially when combined with annealing back to low $10^5$ 4 for stability (Lyle et al., 26 Jul 2025).

Robustness-based interventions attempt to “degrok” models by bringing forward the late robust phase. Perturbation-based training injects Gaussian input or embedding noise with an adaptive scale $10^5$ 5, and an explicit commutativity penalty on modular addition further accelerates the emergence of the correct algebraic invariant. In the reported experiments, both strategies move the test-accuracy jump earlier than baseline training (Tan et al., 2023).

Embedding-centered work suggests two additional practical levers: frequency-aware sampling, to equalize token-update statistics and reduce rare-token stagnation, and embedding-specific learning rates, often with $10^5$ 6 in the reported modular arithmetic tasks. Uniform token coverage improves grokking dynamics relative to skewed sampling, and the adaptive learning-rate ratio derived from bilinear curvature is presented as a general remedy for embedding–downstream coupling in transformer-like systems (AlquBoj et al., 21 May 2025).

Information-theoretic and geometric interventions take a more structural form. Moderate weight decay or high initialization can induce an earlier, stronger emergent-synergy phase in modular addition, whereas extremely high weight decay or extreme initialization suppresses synergy and prevents grokking. Isolating synergistic sub-networks identified during the emergent phase can yield similar or faster generalization than training the full network (Clauw et al., 2024). Entropy-aware optimization via WanD removes the prolonged plateau altogether in modular addition, while in the Ising classifier mild dropout, reduced initialization scale, larger weight decay, or higher softmax temperature shorten or eliminate the delayed transition by preventing or accelerating condensation into a sparse subnetwork (Zhang et al., 16 May 2025, Hutchison et al., 29 Oct 2025).

Open problems remain substantial. The literature provides multiple mechanistic accounts—lazy-to-rich feature learning, norm- and robustness-based explanations, subclass-level distribution shift, phase-transition formalisms, glassy relaxation, metastable escape, entanglement reorganization, and neural-collapse dynamics—but no single theory presently subsumes all regimes. Predicting $10^5$ 7 in advance remains difficult; several papers explicitly identify this as unresolved. Architectural sensitivity is another persistent issue: LayerNorm can block pure ELR-based induction unless attention-input norms are controlled, MLPs may or may not grok depending on embeddings and initialization, and larger-scale language or multimodal continual-learning settings remain largely prospective rather than settled. This suggests that grokking is best understood not as a single mechanism with one universal diagnostic, but as a family of delayed-reorganization phenomena whose common signature is a prolonged separation between fitting and the eventual acquisition of population-level structure (Carvalho et al., 3 Feb 2025, Lyle et al., 26 Jul 2025, Sakamoto et al., 25 Sep 2025).