---
title: Anti-Grokking in Neural Networks
url: https://www.emergentmind.com/topics/anti-grokking
type: topic
---

# Anti-Grokking in Neural Networks

Anti-grokking refers to regimes and interventions in neural network training that (a) prevent or arrest the widely observed delayed transition from memorization to generalization known as “grokking,” and/or (b) describe a late-stage generalization collapse that follows successful grokking, where the model suddenly loses test accuracy while maintaining perfect training accuracy. This phenomenon is quantitatively detected and analyzed using spectral, geometric, and optimization-theoretic diagnostics, and can be mitigated or induced through principled manipulations of sampling, learning rates, regularization, or specialized optimizers. Anti-grokking has been identified and studied in multilayer perceptrons (MLPs), transformers, ridge regression, and even large-scale language models.

## 1. Phenomenology and Definition

Grokking refers to the prolonged lag between attaining perfect training accuracy and achieving high test accuracy during neural network training. Classic grokking phases are:
- **Pre-grokking**: train accuracy →100%, test accuracy at chance.
- **Grokking**: delayed, abrupt jump of test accuracy to near perfect.
- **Anti-grokking**: after extended training beyond the grokking point, test accuracy collapses back towards chance even as training accuracy remains perfect (“generalization collapse” or “overfitting collapse”) [2506.04434, 2602.02859, 2605.12394].

The anti-grokking phase is mechanistically and empirically distinct from pre-grokking: it signals catastrophic overfitting, marked by unique spectral signatures and not simply by sustained overfitting.

## 2. Spectral Diagnostics: Correlation Traps and Heavy-Tailed Metrics

The operational identification of anti-grokking employs tools from random matrix theory (RMT) and heavy-tailed self-regularization (HTSR):
- **Correlation Traps**: large eigenvalues in the shuffled (entry-wise randomly permuted) weight matrix covariance that lie above the Marchenko–Pastur (MP) bulk edge [2602.02859, 2605.12394]. These arise only during anti-grokking, not during memorization or grokking.
- **HTSR α Metric**: For each layer, the empirical spectral density (ESD) of the weight covariance is fitted to a power-law tail, ρ_emp(λ) ∼ λ^(-α). Well-generalized models converge to α ≈ 2. A sharp drop α<2 precedes and uniquely signals anti-grokking [2506.04434, 2602.02859].
- **Trap Counting and Localization**: Benign and harmful traps can be distinguished with data-free ablations using Jensen–Shannon divergence on output distributions, applied to random inputs [2605.12394].

Alternative metrics (ℓ² norm, activation sparsity, absolute weight entropy, local circuit complexity) are insensitive to anti-grokking and only track pre-grok and grok transitions [2506.04434, 2602.02859].

| Phase           | α (HTSR exp.) | Correlation Traps | Generalization      |
|-----------------|:-------------:|:-----------------:|--------------------|
| Pre-grokking    | > 5           | 0                 | No                 |
| Grokking        | ≈2–4          | 0                 | Yes                |
| Anti-grokking   | < 2           | >0                | Collapsing         |

Empirical studies confirm these patterns in MNIST MLPs, modular arithmetic transformers, and large language models (gpt-oss-20B/120B) [2602.02859, 2605.12394].

## 3. Mechanistic Explanations

Several mechanisms contribute to grokking and anti-grokking, depending on architecture and optimization:
- **Embedding Dynamics and Bilinear Coupling**: In architectures with embedding layers, rare tokens (low frequency p_i) stagnate due to sparse updates and weight decay, slowing test accuracy progress. Bilinear forms (e.g., z=WE x) create saddle points due to ±σ_i(E)σ_j(W) Hessian eigenvalues, increasing sensitivity to initialization and curvature [2505.15624].
- **LU Mechanism and Goldilocks Zone**: The mismatch between training and test losses as a function of weight norm (L(w), U(w)) explains the lag as the trajectory shrinks from high-norm (memorized, poor generalization) to “Goldilocks” norm w_c (optimal generalization). Strong regularization or initialization in this zone eliminates grokking [2210.01117].
- **Glass Relaxation Analogy**: Grokking as “computational glass relaxation,” with rapid quench into a non-equilibrium (memorizing) state, followed by slow relaxation into a generalization (high-entropy) regime [2505.11411].

During anti-grokking, the spectral structure evolves towards very heavy-tailed (α<2), and layerwise singular vectors localize to prototype structures, signaling memorization and catastrophic forgetting [2602.02859].

## 4. Algorithmic and Optimization Interventions

Diverse interventions have been demonstrated to achieve anti-grokking (preventing the classical grokking lag, or mitigating anti-grokking collapse) across architectures.

**4.1 Spectral and Gradient Methods**
- **Egalitarian Gradient Descent (EGD)**: Replaces raw gradients with a spectrally “whitened” F^{-1/2}G step, equalizing gradient progress along all principal directions and eliminating ill-conditioning. Removes the grokking plateau and speeds up convergence O(1/ε) → O(1) [2510.04930].
- **Adaptive Learning Rate Ratios**: Set embedding and downstream learning rates so η_E/η_W ∝ (σ_max(E)/σ_max(W))·(f_W/f_E), balancing the bilinear coupling and update frequencies to remove grokking lag [2505.15624].

**4.2 Sampling and Regularization**
- **Frequency-Aware Sampling**: Adjusts token selection probabilities to minimize variance in embedding gradients, up-weighting rare tokens and reducing stagnation [2505.15624].
- **Weight Decay and Data Size**: Strong weight decay (λ, γ), more data, or initialization within the Goldilocks zone (w_0 ≈ w_c) eliminates the delay [2210.01117, 2601.19791].
- **Explicit Spherical Constraints**: Hard projection of weights onto ∥w∥_2 = w_c ensures immediate generalization [2210.01117].

**4.3 Entropy-Driven and Landscape Methods**
- **WanD Optimizer (Wang-Landau Dynamics)**: Stochastic optimizer that constructs a bias potential −S(E) (estimate of negative entropy) over the loss coordinate, flattening the loss landscape and steering into high-entropy, generalizing solutions—eliminating the grokking plateau entirely [2505.11411].
- **Commutator Defect Suppression**: Monitoring and suppressing the commutator defect (a geometric curvature signature of non-commuting gradient steps) with hard or soft projections can delay or prevent grokking and generalization collapse [2602.16967].

## 5. Quantitative and Empirical Evidence

Anti-grokking and its mitigation have been validated across settings:
- **MNIST, Modular Arithmetic, and Synthetic Tasks**: Anti-grokking arises after 10⁶–10⁷ steps in 3-layer MLPs and transformers with small datasets and zero or weak weight decay [2602.02859, 2605.12394, 2506.04434]. Layer α drops below 2 and traps appear precisely as test accuracy collapses.
- **Transformer Architectures**: Frequency-aware sampling and embedding-specific learning rates accelerate generalization by an order of magnitude; suppressing commutator defect delays or eliminates generalization [2505.15624, 2602.16967].
- **Large Language Models**: Correlation traps and α<2 signatures are detected in foundation models, raising concerns about latent anti-grokking pathologies [2602.02859, 2605.12394].
- **Ridge Regression**: Explicit bounds show that λ ≳ λ_min⁺(ΦᵀΦ)/n is sufficient to eliminate grokking delays and force coincident decay of training and generalization error [2601.19791].

## 6. Monitoring and Practical Guidelines

Robust, data-free early-warning signals and best practices for anti-grokking include:
- **WeightWatcher**: Track per-layer α and trap counts periodically. α<2 or nonzero trap count signals imminent collapse [2602.02859, 2506.04434, 2605.12394].
- **Mitigation and Control**:
  - Increase or enforce weight decay (γ, λ).
  - Initiate training within or project onto the Goldilocks norm zone.
  - Early-stopping at sustained high test accuracy or α≈2 without traps.
  - Reduce learning rates or enforce spectral norm constraints on problematic layers.
  - Integrate spectral diagnostics into CI pipelines to monitor for emergent anti-grokking in large-scale pretraining [2602.02859].

## 7. Implications, Open Problems, and Extensions

- **Catastrophic Forgetting and Prototype Memorization**: Anti-grokking is mechanistically distinct from pre-grokking. It involves localization of singular vectors to training prototypes and large rank-1 perturbations, disrupting generalized solutions [2602.02859].
- **Universality and Scaling**: Current evidence spans MLPs, transformers, and linear models; detection in LLMs suggests broad applicability. Scaling of interventions to production-scale domains remains to be fully validated [2602.16967].
- **Complementarity of Diagnostics**: Spectral (α, traps), geometric (commutator defect), and entropy (WanD) perspectives provide cross-validating, architecture-agnostic signals.
- **Regularization without Test Data**: Both α and correlation trap metrics provide genuinely data-free early warnings, enabling safety and reliability monitoring without peeking at held-out sets [2605.12394, 2602.02859].
- **Open Tuning Problems**: Suppression interventions may prevent both undesirable collapse and useful late-stage refinements; balancing tradeoffs is an unsolved challenge [2602.16967].

**In summary, anti-grokking encompasses both the quantitative detection of late-stage generalization collapse and the principled elimination of grokking-related delays. Its study brings together spectral theory, optimization geometry, and entropy-driven landscape modeling, yielding practical diagnostics and interventions for robust generalization and overfitting avoidance in modern neural architectures.**

Source: https://www.emergentmind.com/topics/anti-grokking