Grokking: Delayed Generalization in Neural Networks

Updated 10 December 2025

Grokking is a delayed generalization phenomenon in neural networks, marked by an abrupt shift from memorization to robust performance.
It unfolds in distinct stages: a memorization phase with near-zero training loss, a stagnant plateau phase, and a rapid transition phase indicating circuit reorganization.
Metrics such as synergy, activation sparsity, and weight entropy provide practical signals to predict and control this emergent phase transition.

Grokking is a delayed generalization phenomenon in over-parameterized neural networks, characterized by a network achieving near-perfect training accuracy while maintaining near-random test performance over an extended training plateau, before abruptly transitioning to high test accuracy after many further training epochs. This discrete transition from memorization to generalization is not a smooth or gradual improvement but a sharply defined, emergent event that resists explanation by classical generalization or overfitting paradigms (Clauw et al., 16 Aug 2024, Golechha, 21 May 2024).

1. Precise Characteristics and Empirical Dynamics

Grokking typically unfolds in three distinct training phases:

Memorization Phase: The network fits the training set, rapidly lowering training loss to near zero. The test loss or error, however, remains at or near the random baseline, and internal representations are dominated by redundant, isolated features (Clauw et al., 16 Aug 2024, Golechha, 21 May 2024).
Plateau Phase: Even as the model continues to train, test performance remains stagnant. Little change is observed in aggregate loss curves, and the internal structure continues to evolve without apparent gains in generalization.
Grokking (Transition) Phase: Test accuracy or validation loss suddenly improves, typically within a small number of epochs, matching the previously high training performance. The underlying network reorganizes, often forming compact sub-networks or circuits characterized by new internal feature cooperation (Clauw et al., 16 Aug 2024, Gromov, 2023).

The sharpness of this transition can be quantified by fitting S-shaped error functions to training and test accuracy curves and measuring parameters such as the relative delay and the slope at the transition point (Miller et al., 14 Feb 2024). The grokking delay is highly sensitive to hyperparameters, training data fraction, and regularization strength.

2. Mechanistic Theories: Synergy, Emergence, and Phase Transition

A central mechanistic insight is that grokking corresponds to a bona fide emergent phase transition in neural network training (Clauw et al., 16 Aug 2024, Žunkovič et al., 2022, Hutchison et al., 29 Oct 2025):

Order Parameter: The information-theoretic measure of synergy among neural units emerges as a rigorous order parameter. Synergy, defined as the extra mutual information gained when considering the collective output of a group of neurons versus their individual contributions, remains low during memorization and then spikes sharply at the transition, recapitulating phase-transition dynamics in physics (Clauw et al., 16 Aug 2024).
Distinct Phases: Information-theoretic progress measures partition training into Feature Learning (redundancy-dominated), Emergence (rapid synergy gain), Divergence (overfit/decay), Delayed Emergence (second synergy surge), and Decoupling/Compression (redundancy rises, synergy falls, and generalizing circuits are pruned) (Clauw et al., 16 Aug 2024).
Critical Point Control: Weight decay and initialization scale shift the transition point. High weight decay smooths the transition and reduces delay (or can eliminate grokking entirely), while inappropriate initialization scales can prevent the phase transition (Clauw et al., 16 Aug 2024).
Predictive Signatures: Early-training nontrivial peaks in synergy or analogous progress measures robustly predict the impending occurrence of grokking (Clauw et al., 16 Aug 2024, Notsawo et al., 2023).

3. Progress Measures and Generalization Signals

Traditional heuristics such as the $L_2$ weight norm fail to universally explain or predict grokking. Instead, robust progress measures have been introduced (Golechha, 21 May 2024):

Activation Sparsity: The fraction of "off" neurons. Grokking is foreshadowed by rising or plateauing sparsity prior to the generalization jump.
Absolute Weight Entropy: A Shannon-entropy–inspired measure on absolute weight magnitudes. A drop in entropy marks the onset of generalization, even when weight norm evolves oppositely.
Approximate Local Circuit Complexity: Measured as the KL divergence between output logits before and after random local weight ablation. A decrease in this metric signals that the learned circuit is becoming robust and lower-complexity, immediately preceding grokking.

These measures exhibit precursor signatures—drops, plateaus, or surges—before the observed generalization transition and are consistent across real-world tasks and architectures, in contrast to weight norm–based criteria.

4. Mathematical and Algorithmic Underpinnings

From a mathematical perspective, grokking is linked to the learning dynamics of gradient-based methods in high-dimensional, non-convex loss landscapes:

Gradient Timescales: There is a separation between fast-varying modes (responsible for overfitting) and slow-varying modes (responsible for eventual generalization). Spectral filtering of gradients, as in the Grokfast algorithm, can amplify the slow, generalization-inducing components and accelerate the transition by more than 50× (Lee et al., 30 May 2024).
Ill-conditioned Optimization: Vanilla SGD proceeds at asymmetric rates along the principal directions of the Fisher or empirical Hessian. Small singular-value directions evolve slowly, creating the long generalization plateau. Egalitarian Gradient Descent equalizes per-direction update rates and can virtually eliminate the grokking delay (Pasand et al., 6 Oct 2025).
Theoretical Models: Exact phase-transition analogies and solvable models have been furnished for linear estimators, perceptrons on local rules, and glassy systems, revealing that grokking time diverges as one approaches data-complexity thresholds, and that it often realizes a second-order transition with analytically calculable critical exponents and distributions (Žunkovič et al., 2022, Hutchison et al., 29 Oct 2025, Zhang et al., 16 May 2025).

5. Controlling, Predicting, and Diagnosing Grokking

Grokking is modulated by both data and architectural factors:

Regularization Knobs: Varying weight decay, learning rate, and initialization can control the order and occurrence of the phase transition. Excessive regularization or extreme initialization eliminates or delays grokking (Clauw et al., 16 Aug 2024, Golechha, 21 May 2024).
Early Diagnosis: Monitoring progress measures (synergy, entropy, complexity) within tens of initial epochs provides reliable early-warning signals of whether a model will eventually grok if trained exhaustively (Clauw et al., 16 Aug 2024, Notsawo et al., 2023).
Distributional Factors: Recent statistical perspectives demonstrate that even mild train/test distribution shift (e.g., imbalanced class or subclass sampling) is a sufficient and necessary driver for grokking. In this view, small-sample regimes or subclass imbalances do not cause grokking directly, but serve as mechanisms for introducing the critical data distribution mismatch needed to trigger late generalization (Carvalho et al., 3 Feb 2025).
Practical Measures: Progress measures may, in future, be used as regularizers to directly control the timing or guarantee the occurrence of grokking (Golechha, 21 May 2024).

6. Structural and Circuit-Level Interpretation

Grokking corresponds not only to an abrupt change in global accuracy but often to the mechanistic emergence of specific internal circuits or representations:

Interpretability: On algorithmic or group-theoretic tasks (e.g., modular arithmetic), the transition is marked by the sudden acquisition of Fourier-based feature maps or trigonometric circuits, as revealed by analytical and empirical decomposition of the learned weight matrices and activations (Gromov, 2023, Furuta et al., 26 Feb 2024).
Structural Reorganization: PCA and sparsity analyses demonstrate that, during grokking, dense networks reconfigure into sparse, core sub-networks that align with the dataset's invariants or symmetries (Hutchison et al., 29 Oct 2025). In Ising model classifiers, grokking was shown to coincide with a collapse in parameter rank, emergence of distinct class clusters in latent space, and pruning of excess network connectivity.
Information-Theoretic Metrics: Synergy and redundancy, as well as perturbed mutual information metrics, may serve both as signatures of and practical diagnostics for the formation of generalizing structures inside trained models (Clauw et al., 16 Aug 2024, Tan et al., 2023).

7. Broader Implications and Future Directions

Grokking is not restricted to toy or synthetic tasks but arises in real-world classification and reasoning, including language, vision, and graph-based datasets (Golechha, 21 May 2024, Lee et al., 30 May 2024, Abramov et al., 29 Apr 2025). It challenges prevailing expectations of monotonic generalization and reveals the need for new progress measures beyond classical loss, accuracy, or weight-norm trajectories. The phenomenon invites further inquiries into:

The detailed geometry and thermodynamics of non-convex optimization in deep learning (Kozyrev, 17 Dec 2024, Zhang et al., 16 May 2025).
A comprehensive theory relating emergence of generalization to collective, higher-order interactions among network components (Clauw et al., 16 Aug 2024).
Automated early-stopping criteria or curricula guided by information-theoretic or circuit-level progress indicators (Clauw et al., 16 Aug 2024, Golechha, 21 May 2024, Notsawo et al., 2023).
Extensions to deeper, larger-scale models and the mechanism's persistence or modification under more complex data and tasks (Carvalho et al., 3 Feb 2025, Golechha, 21 May 2024, Qiye et al., 14 Dec 2024).

The theoretical and empirical research converges on the view of grokking as a prototypical emergent phase transition, where delayed collective reorganization, driven by synergy and circuit formation under the influence of regularization and data geometry, accounts for the abrupt leap from rote memorization to core generalization—a phenomenon both fascinating for theory and actionable for practice (Clauw et al., 16 Aug 2024).