---
title: Grokking Phenomenon in Neural Networks
url: https://www.emergentmind.com/topics/grokking-phenomenon
type: topic
---

# Grokking Phenomenon in Neural Networks

Grokking is a counter-intuitive phenomenon in neural network training, characterized by prolonged overfitting—training loss reaches (near) zero while test loss remains high—followed by an abrupt and dramatic improvement in test performance after many additional optimization steps. This delayed generalization emerges across a variety of architectures (MLPs, transformers, CNNs), tasks (algorithmic, image, parity, group composition), and optimization regimes. Unlike classical overfitting scenarios, where excessive training leads only to degraded generalization, the grokking regime features a clear two-phase training curve: an early memorization plateau and a sudden "grok" event where the model internalizes the underlying rule or structure, enabling robust out-of-sample prediction. Grokking has proven foundational for analyzing deep-network dynamics, algorithmic learning, generalization, and serves as a touchstone for the development of new complexity and robustness metrics, optimizer modifications, and theoretical frameworks.

## 1. Empirical Phenomenology and Operational Definition

Grokking manifests as a sharp temporal separation between training set memorization and test set generalization. Let $A_{\mathrm{train}}(t)$ and $A_{\mathrm{test}}(t)$ denote the training and test accuracies at time (or iteration) $t$, with loss functions $L_{\mathrm{train}}(t)$ and $L_{\mathrm{test}}(t)$. In the canonical grokking behavior one observes:
- $A_{\mathrm{train}}(t) \to 1$ (or $L_{\mathrm{train}}(t) \to 0$) for $t \gg t_0$,
- $A_{\mathrm{test}}(t) \approx A_{\mathrm{chance}}$ (e.g., $1/\mathrm{\#classes}$) for $t_0 \ll t < t_g$,
- then, at $t \approx t_g$, $A_{\mathrm{test}}(t)$ rapidly increases to $\sim 1$ (or $L_{\mathrm{test}}(t)$ suddenly collapses).

This two-phase dynamic is robust across architectures, including two-layer MLPs, deep MLPs, transformers, CNNs, and graph neural networks, and is observed in both synthetic (modular arithmetic, parity, group operations) and real-world datasets (MNIST, CIFAR-10, CIFAR-100, Imagenette) [2405.17479, 2405.19454, 2402.15555, 2505.11411]. Grokking plateaus are quantifiable via the time gap $\Delta t = t_{\mathrm{grok}} - t_{\mathrm{mem}}$, where $t_{\mathrm{mem}}$ is the step at which train accuracy saturates.

## 2. Core Mechanistic Theories

Multiple theoretical frameworks have been advanced to explain grokking. The major lines are:

1. **Frequency-Domain Misalignment:** Grokking is driven by a two-phase spectral learning process. Neural networks initially fit spurious low-frequency components caused by non-uniform or undersampled training data (spectral aliasing), leading to rapid train loss decrease but poor test performance. Only after prolonged training do the networks fit the true, typically higher-frequency, components required for generalization, aligning spectral modes between train and test sets [2405.17479]. This explanation is corroborated by Fourier analysis on sinusoidal, Boolean, and image tasks.

2. **Generalization–Compression Phases:** Networks transition from a high-complexity, memorizing solution to an information-compressed, generalizing one. Metrics such as Linear Mapping Number (LMN) and an algorithmic rate–distortion complexity bound (Kolmogorov-inspired) show a rise during memorization and a sharp fall at grokking, with the test loss linearly tracking model complexity during this compression window [2310.05918, 2412.09810].

3. **Kernel-to-Feature-Learning Transitions:** Early in training, wide networks behave as kernel predictors (NTK or GP regime), which interpolate training data but require an $\Omega(1)$ fraction of the input space to generalize (impossible with small samples). Grokking coincides with the escape from the kernel/lazy regime into a "rich," feature-learning regime, often mediated by $\ell_\infty$-norm (or margin) bias from weight decay or implicit regularization, enabling efficient generalization even from sparse examples [2407.12332, 2311.18817].

4. **Glassy Relaxation Analogy:** Training is mapped onto a non-equilibrium glass relaxation. The network rapidly descends into a "memorization basin" (low-loss, low-entropy state), from which it slowly relaxes into a high-entropy, generalizing basin. There is no entropy barrier in the transition—the process is barrier-free relaxation rather than a first-order phase transition [2505.11411].

5. **Robustness and Regularization:** Decay of the $\ell_2$ weight norm increases the "radius of robustness," causing the model's predictions to become stable under perturbation and pulling test points inside the effective decision margin, which triggers sudden generalization. Input/noise augmentation and explicit group invariance regularizers (e.g., commutativity in modular addition) can "de-grok" (accelerate) this transition [2311.06597].

6. **Spectral Bottlenecks and Optimizer Effects:** Grokking plateaus arise from extreme spectral imbalance in gradient dynamics—fast modes (principal directions) are learned quickly, but "slow modes" critical for generalization evolve orders of magnitude more slowly. Modifying the optimizer to equalize spectral speeds (e.g., Egalitarian Gradient Descent) or amplify slow gradient components (Grokfast algorithm) can dramatically shorten or eliminate grokking delays [2510.04930, 2405.20233].

7. **Phase Transition Perspective:** Statistical mechanics approaches model grokking as a first-order phase transition, with order parameters describing feature alignment or kernel eigenstructure crossing a critical point, leading to emergent generalization in a mixed-feature phase [2310.03789].

8. **Statistical and Data Distribution View:** Grokking is associated with a distribution shift between training and test datasets, most starkly visible under sub-category imbalance or missing subclasses. Generalization is delayed until regularization (direct or implicit) forces the boundary to align with under-represented or absent test regions [2502.01774].

## 3. Experimental Paradigms and Diagnostic Metrics

Empirical studies span tasks and architectures:
- **Algorithmic:** Modular addition/multiplication, parity, group composition, XOR. Highly controlled, favor delayed generalization [2407.12332, 2310.03789, 2310.16441].
- **Real-World Datasets:** MNIST, CIFAR-10/100, Imagenette, QM9, IMDb [2405.19454, 2402.15555, 2405.20233].
- **Benchmarks:** Small subsets of data, biased sampling, or deliberate class imbalance to trigger distribution shifts [2502.01774].

Quantitative and diagnostic tools include:

| Metric                 | Purpose/Insight                        | Typical Usage              |
|------------------------|----------------------------------------|----------------------------|
| LMN, Rate–Distortion   | Model intrinsic complexity/compression | Tracks the compression phase and predicts grokking [2310.05918, 2412.09810] |
| Local Complexity $C_{\mathrm{loc}}$ | Spline-region density / partition geometry | Diagnoses grokking, delayed robustness, and region migration in input space [2402.15555] |
| Robustness metrics (PE/PMI, MID/ED)   | Predict grokking, quantify input–output stability | Early indicators, can accelerate transition [2311.06597] |
| NTK/Feature Covariance Rotation       | Marks kernel–to–feature transitions     | Direct visualization of representation learning [2407.12332, 2507.20057] |
| Spectral Oscillation (Fourier loss curve) | Early prediction of grokking           | Allows pruning hyperparameter search [2306.13253] |
| Sharpness/Gap Parametrization         | Quantifies sharpness of accuracy transition | Enables comparison of grokking gap vs. transition sharpness [2402.08946] |
| Feature Rank Collapse                 | Detects multistage generalization, tunnel effect | Superior to weight norm as predictor in deep MLPs [2405.19454] |

## 4. Algorithmic and Optimization Strategies for Accelerating or Eliminating Grokking

The slow-to-emerge generalization of grokking can be mitigated or eliminated via several algorithmic interventions:

- **Spectral Equalization/Gradient Manipulation:** Egalitarian GD or Grokfast precondition the gradient, amplifying slow directions to equalize learning speeds and collapse plateaus. These approaches excel in parity, modular arithmetic, and other settings sensitive to gradient spectrum [2510.04930, 2405.20233].
- **High-Entropy–Seeking Optimizers:** WanD (Wang–Landau MD in parameter space) samples parameter regions of high entropy at fixed loss, favoring generalizing solutions and bypassing the delayed glassy relaxation phase [2505.11411].
- **Norm Regularization and Robustness Augmentation:** Weight decay or explicit robustness induction (Gaussian input noise, Jacobian/Lipschitz penalties, group-theory regularizers) can accelerate grokking, often by enforcing the necessary algebraic invariances to support generalization [2311.06597].
- **Numerical Remedies:** StableMax activation and perpendicular-gradient optimizers prevent softmax collapse and naive loss minimization, confronting the numerical "edge-of-stability" issues that can prevent generalization without regularization [2501.04697].
- **Knowledge Distillation:** Transferring knowledge from a model that has already grokked on a related (or even different) distribution can induce grokking in data-scarce or distribution-shifted settings, reducing the critical data threshold and mitigating catastrophic forgetting in continual learning [2511.04760].
- **ELR Re-Warming:** Periodic effective learning-rate resets (e.g., Normalize-and-Project or cyclical learning rate schedules) re-initiate rich feature-learning, enabling on-demand grokking and overcoming primacy bias in online or nonstationary tasks [2507.20057].

## 5. Connections to Complexity, Compression, and Generalization Theory

Grokking is now firmly situated at the intersection of dynamic model complexity, implicit regularization, and information-theoretic compression. The consensus across approaches is that:
- Early solutions after training set interpolation are high complexity (e.g., large LMN, high intrinsic code length), corresponding to lookup-table like memorization.
- The compression phase, sometimes observed as a double-descent in feature rank or complexity, culminates in the sudden collapse to a low-rank, low-entropy, highly compressible representation—a precondition for robust generalization [2310.05918, 2412.09810, 2405.19454].
- Explicit rate–distortion or MDL principles can yield PAC-Bayes–style generalization guarantees tied closely to empirical complexity estimates [2412.09810].

Notably, in fully linear or kernelized settings, grokking can arise purely as a measurement artifact: smooth loss curves pass a non-linear accuracy threshold later for test than for train data, without any change in the solution's nature [2310.16441].

## 6. Practical Guidance, Limitations, and Open Questions

Practical recommendations include:
- ***Early detection:*** Monitor information-theoretic, robustness, or complexity metrics (e.g., PE/PMI, LMN, feature rank) to predict and detect grokking transitions earlier than test accuracy would reveal [2311.06597, 2310.05918, 2405.19454].
- ***Avoidance vs. Exploitation:*** To eliminate undesirable grokking plateaus, use footprint-efficient optimizers, high-entropy trajectory sampling, or explicit regularization; to harness grokking for structured algorithmic learning, maintain moderate data sparsity, small weight decay, and allow delayed onset [2510.04930, 2405.20233].
- ***Complexity regularization:*** Spectral entropy regularizers systematically induce easier compression and more predictable grokking [2412.09810].
- ***Continual learning and transfer:*** Knowledge distillation and re-warming protocols overcome critical data thresholds, enable transfer, and prevent forgetting [2511.04760, 2507.20057].
- ***Statistical phenomenon, not just sparsity:*** Grokking is fundamentally tied to distribution shift, not just data sparsity or high regularization; richness or explicit equivariance in class structure can substitute for sample presence in accelerating grokking [2502.01774].

Open questions encompass:
- The universality and precise criticality of observed phase transitions—barrier-free (glassy) or sharp (first-order);
- The minimal sufficient conditions for grokking in high-data, low-regularization, or practical (non-algorithmic) domains;
- The interactions between numerical phenomena (edge-of-stability, floating-point effects) and representational learning;
- Whether more powerful hybrid regularization, optimizer, or architectural methods can arbitrarily schedule or eliminate grokking;
- The optimality of sharpness/gap metrics for comparing grokking across diverse models and tasks [2402.08946].

Grokking remains a central diagnostic and theoretical tool for interrogating the transition from memorization to generalization in deep networks, with implications for representation learning, optimization, robustness, and complexity theory across the entire spectrum of modern machine learning.

Source: https://www.emergentmind.com/topics/grokking-phenomenon