Anti-Grokking in Neural Networks

• Anti-grokking is a regime where neural networks, after achieving perfect training accuracy, experience a sudden collapse in test performance due to catastrophic overfitting.
• Spectral diagnostics like the HTSR α metric and correlation traps identify anti-grokking by signaling α drops below 2 and the emergence of significant eigenvalue outliers.
• Mitigation strategies include tailored learning rates, regularization, frequency-aware sampling, and specialized optimizers to stabilize generalization across various architectures.

Anti-grokking refers to regimes and interventions in neural network training that (a) prevent or arrest the widely observed delayed transition from memorization to generalization known as “grokking,” and/or (b) describe a late-stage generalization collapse that follows successful grokking, where the model suddenly loses test accuracy while maintaining perfect training accuracy. This phenomenon is quantitatively detected and analyzed using spectral, geometric, and optimization-theoretic diagnostics, and can be mitigated or induced through principled manipulations of sampling, learning rates, regularization, or specialized optimizers. Anti-grokking has been identified and studied in multilayer perceptrons (MLPs), transformers, ridge regression, and even large-scale LLMs.

1. Phenomenology and Definition

Grokking refers to the prolonged lag between attaining perfect training accuracy and achieving high test accuracy during neural network training. Classic grokking phases are:

• Pre-grokking: train accuracy →100%, test accuracy at chance.
• Grokking: delayed, abrupt jump of test accuracy to near perfect.
• Anti-grokking: after extended training beyond the grokking point, test accuracy collapses back towards chance even as training accuracy remains perfect (“generalization collapse” or “overfitting collapse”) (Prakash et al., 4 Jun 2025, Prakash et al., 2 Feb 2026, Prakash et al., 12 May 2026).

The anti-grokking phase is mechanistically and empirically distinct from pre-grokking: it signals catastrophic overfitting, marked by unique spectral signatures and not simply by sustained overfitting.

2. Spectral Diagnostics: Correlation Traps and Heavy-Tailed Metrics

The operational identification of anti-grokking employs tools from random matrix theory (RMT) and heavy-tailed self-regularization (HTSR):

• Correlation Traps: large eigenvalues in the shuffled (entry-wise randomly permuted) weight matrix covariance that lie above the Marchenko–Pastur (MP) bulk edge (Prakash et al., 2 Feb 2026, Prakash et al., 12 May 2026). These arise only during anti-grokking, not during memorization or grokking.
• HTSR α Metric: For each layer, the empirical spectral density (ESD) of the weight covariance is fitted to a power-law tail, ρ_emp(λ) ∼ λ^-α. Well-generalized models converge to α ≈ 2. A sharp drop α<2 precedes and uniquely signals anti-grokking (Prakash et al., 4 Jun 2025, Prakash et al., 2 Feb 2026).
• Trap Counting and Localization: Benign and harmful traps can be distinguished with data-free ablations using Jensen–Shannon divergence on output distributions, applied to random inputs (Prakash et al., 12 May 2026).

Alternative metrics (ℓ² norm, activation sparsity, absolute weight entropy, local circuit complexity) are insensitive to anti-grokking and only track pre-grok and grok transitions (Prakash et al., 4 Jun 2025, Prakash et al., 2 Feb 2026).

Phase	α (HTSR exp.)	Correlation Traps	Generalization
Pre-grokking	> 5	0	No
Grokking	≈2–4	0	Yes
Anti-grokking	< 2	>0	Collapsing

Empirical studies confirm these patterns in MNIST MLPs, modular arithmetic transformers, and LLMs (gpt-oss-20B/120B) (Prakash et al., 2 Feb 2026, Prakash et al., 12 May 2026).

3. Mechanistic Explanations

Several mechanisms contribute to grokking and anti-grokking, depending on architecture and optimization:

• Embedding Dynamics and Bilinear Coupling: In architectures with embedding layers, rare tokens (low frequency p_i) stagnate due to sparse updates and weight decay, slowing test accuracy progress. Bilinear forms (e.g., z=WE x) create saddle points due to ±σ_i(E)σ_j(W) Hessian eigenvalues, increasing sensitivity to initialization and curvature (AlquBoj et al., 21 May 2025).
• LU Mechanism and Goldilocks Zone: The mismatch between training and test losses as a function of weight norm (L(w), U(w)) explains the lag as the trajectory shrinks from high-norm (memorized, poor generalization) to “Goldilocks” norm w_c (optimal generalization). Strong regularization or initialization in this zone eliminates grokking (Liu et al., 2022).
• Glass Relaxation Analogy: Grokking as “computational glass relaxation,” with rapid quench into a non-equilibrium (memorizing) state, followed by slow relaxation into a generalization (high-entropy) regime (Zhang et al., 16 May 2025).

During anti-grokking, the spectral structure evolves towards very heavy-tailed (α<2), and layerwise singular vectors localize to prototype structures, signaling memorization and catastrophic forgetting (Prakash et al., 2 Feb 2026).

4. Algorithmic and Optimization Interventions

Diverse interventions have been demonstrated to achieve anti-grokking (preventing the classical grokking lag, or mitigating anti-grokking collapse) across architectures.

4.1 Spectral and Gradient Methods

• Egalitarian Gradient Descent (EGD): Replaces raw gradients with a spectrally “whitened” F^{-1/2}G step, equalizing gradient progress along all principal directions and eliminating ill-conditioning. Removes the grokking plateau and speeds up convergence O(1/ε) → O(1) (Pasand et al., 6 Oct 2025).
• Adaptive Learning Rate Ratios: Set embedding and downstream learning rates so η_E/η_W ∝ (σ_max(E)/σ_max(W))·(f_W/f_E), balancing the bilinear coupling and update frequencies to remove grokking lag (AlquBoj et al., 21 May 2025).

4.2 Sampling and Regularization

• Frequency-Aware Sampling: Adjusts token selection probabilities to minimize variance in embedding gradients, up-weighting rare tokens and reducing stagnation (AlquBoj et al., 21 May 2025).
• Weight Decay and Data Size: Strong weight decay (λ, γ), more data, or initialization within the Goldilocks zone (w_0 ≈ w_c) eliminates the delay (Liu et al., 2022, Xu et al., 27 Jan 2026).
• Explicit Spherical Constraints: Hard projection of weights onto ∥w∥_2 = w_c ensures immediate generalization (Liu et al., 2022).

4.3 Entropy-Driven and Landscape Methods

• WanD Optimizer (Wang-Landau Dynamics): Stochastic optimizer that constructs a bias potential −S(E) (estimate of negative entropy) over the loss coordinate, flattening the loss landscape and steering into high-entropy, generalizing solutions—eliminating the grokking plateau entirely (Zhang et al., 16 May 2025).
• Commutator Defect Suppression: Monitoring and suppressing the commutator defect (a geometric curvature signature of non-commuting gradient steps) with hard or soft projections can delay or prevent grokking and generalization collapse (Xu, 19 Feb 2026).

5. Quantitative and Empirical Evidence

Anti-grokking and its mitigation have been validated across settings:

• MNIST, Modular Arithmetic, and Synthetic Tasks: Anti-grokking arises after 10⁶–10⁷ steps in 3-layer MLPs and transformers with small datasets and zero or weak weight decay (Prakash et al., 2 Feb 2026, Prakash et al., 12 May 2026, Prakash et al., 4 Jun 2025). Layer α drops below 2 and traps appear precisely as test accuracy collapses.
• Transformer Architectures: Frequency-aware sampling and embedding-specific learning rates accelerate generalization by an order of magnitude; suppressing commutator defect delays or eliminates generalization (AlquBoj et al., 21 May 2025, Xu, 19 Feb 2026).
• LLMs: Correlation traps and α<2 signatures are detected in foundation models, raising concerns about latent anti-grokking pathologies (Prakash et al., 2 Feb 2026, Prakash et al., 12 May 2026).
• Ridge Regression: Explicit bounds show that λ ≳ λ_min⁺(ΦᵀΦ)/n is sufficient to eliminate grokking delays and force coincident decay of training and generalization error (Xu et al., 27 Jan 2026).

6. Monitoring and Practical Guidelines

Robust, data-free early-warning signals and best practices for anti-grokking include:

• WeightWatcher: Track per-layer α and trap counts periodically. α<2 or nonzero trap count signals imminent collapse (Prakash et al., 2 Feb 2026, Prakash et al., 4 Jun 2025, Prakash et al., 12 May 2026).
• Mitigation and Control:
  • Increase or enforce weight decay (γ, λ).
  • Initiate training within or project onto the Goldilocks norm zone.
  • Early-stopping at sustained high test accuracy or α≈2 without traps.
  • Reduce learning rates or enforce spectral norm constraints on problematic layers.
  • Integrate spectral diagnostics into CI pipelines to monitor for emergent anti-grokking in large-scale pretraining (Prakash et al., 2 Feb 2026).

7. Implications, Open Problems, and Extensions

• Catastrophic Forgetting and Prototype Memorization: Anti-grokking is mechanistically distinct from pre-grokking. It involves localization of singular vectors to training prototypes and large rank-1 perturbations, disrupting generalized solutions (Prakash et al., 2 Feb 2026).
• Universality and Scaling: Current evidence spans MLPs, transformers, and linear models; detection in LLMs suggests broad applicability. Scaling of interventions to production-scale domains remains to be fully validated (Xu, 19 Feb 2026).
• Complementarity of Diagnostics: Spectral (α, traps), geometric (commutator defect), and entropy (WanD) perspectives provide cross-validating, architecture-agnostic signals.
• Regularization without Test Data: Both α and correlation trap metrics provide genuinely data-free early warnings, enabling safety and reliability monitoring without peeking at held-out sets (Prakash et al., 12 May 2026, Prakash et al., 2 Feb 2026).
• Open Tuning Problems: Suppression interventions may prevent both undesirable collapse and useful late-stage refinements; balancing tradeoffs is an unsolved challenge (Xu, 19 Feb 2026).

In summary, anti-grokking encompasses both the quantitative detection of late-stage generalization collapse and the principled elimination of grokking-related delays. Its study brings together spectral theory, optimization geometry, and entropy-driven landscape modeling, yielding practical diagnostics and interventions for robust generalization and overfitting avoidance in modern neural architectures.