Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

How to Tame Grokking: Representation Geometry as a Control Signal

Published 13 Jul 2026 in cs.LG | (2607.11666v1)

Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.

Authors (1)

Summary

  • The paper demonstrates that inducing a rapid collapse in representation dimensionality via GeomDR is key to accelerating the grokking transition in neural networks.
  • It details how spectral regularization manipulates hidden representations by constraining covariance spectra, leading to improved generalization across tasks.
  • The work provides empirical evidence from MLPs and Transformers, with quantitative benchmarks highlighting the practical benefits of geometric intervention.

Representation Geometry as a Control Signal for Grokking

Introduction

This work presents a comprehensive study of the relationship between representation geometry—specifically, representation dimensionality—and the phenomenon of grokking in neural networks. Grokking refers to the empirical observation that, in certain algorithmic tasks, neural networks first memorize the training data but only acquire strong generalization capabilities after a prolonged training regime. It has remained unclear whether the delayed onset of generalization is merely correlated with, or mechanistically driven by, changes in the geometry of learned representations. The paper introduces Geometric Dimensionality Regularization (GeomDR), a spectral regularizer targeting the covariance spectra of hidden representations, and demonstrates that manipulating representation geometry provides direct control over the timing and character of grokking across various architectures and synthetic algorithmic tasks.

Grokking and Representation Geometry

Baseline experiments replicate delayed generalization in synthetic modular arithmetic tasks. Trajectories of effective dimensionality and local neighborhood distance through training reveal sharp reorganization phases in the representation space coinciding with the test accuracy transition, particularly in deeper layers. In all cases, dimensionality collapse—a reduction of the effective number of active covariance dimensions—precedes the onset of generalization, suggesting a tight temporal relationship between these geometric signatures and functional grokking. Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1: Baseline learning: effective dimensionality, local neighborhood metric, and accuracy curves, illustrating abrupt geometry transitions immediately prior to test accuracy increase.

Unsmoothed geometric traces show that this is not a slow, continuous collapse; rather, a rapid, punctuated change in the representation metric signals the imminent success in test accuracy. These metrics jointly support the hypothesis that a reduction in the complexity of the internal representation is a necessary precursor for generalization on these tasks.

Geometric Dimensionality Regularization (GeomDR)

GeomDR is a representation-level objective that penalizes covariance outside a user-specified number of dominant principal components in each hidden layer. For a target dimensionality d∗d^*, the spectral regularizer LGeomDR(ℓ)=∑j=d∗+1dμj(ℓ)L_{\mathrm{GeomDR}}^{(\ell)} = \sum_{j=d^*+1}^{d}\mu_j^{(\ell)} suppresses variance in trailing components. The regularizer is ramped in gradually after an initial unconstrained phase, in order to not disrupt the phase of model fitting dynamics when memorizing solutions first emerge. The full loss is L=Ltask+λ(t)LGeomDRL = L_\mathrm{task} + \lambda(t) L_{\mathrm{GeomDR}}.

The regularization schedule and the choice of d∗d^* critically affect the model's learning trajectory. Sweeps over these parameters show robust acceleration in grokking, provided the regularization is neither too strong nor too weak and the target dimensionality is not overly restrictive. Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2: Imposing target dimensionalities via GeomDR leads to immediate and pronounced geometric transitions and substantially accelerated grokking.

Intermediate values for both regularization strength λmax⁡\lambda_{\max} and target dimension d∗d^* (16≤d∗≤6416 \leq d^* \leq 64 for modular addition) yield consistent and large effects, while over-compression (d∗≤8d^* \leq 8) can destabilize convergence. Temporal sweeps demonstrate that GeomDR is most effective when applied after an initial period of memorization but before the transition to rigid, task-agnostic representations. Figure 3

Figure 3: Schedule sweep: intermediate ramp and regularization yield maximal acceleration of grokking; overly aggressive settings can suppress generalization entirely.

Geometric Causality and Mechanistic Implications

Systematic ablations establish that geometric intervention via GeomDR provides causal leverage over the grokking transition. Across all tasks and architectures, collapse in effective dimensionality tightly precedes, and with GeomDR almost coincides with, the transition to test generalization. For MLPs, well-chosen regularizer configurations compress the lag between collapse and grokking from hundreds of thousands to mere thousands of update steps. Figure 4

Figure 4: Intermediate and large d∗d^* values accelerate grokking; aggressive compression slows and destabilizes learning.

Figure 5

Figure 5: Representation collapse consistently precedes the grokking step; GeomDR narrows the collapse–grokking lag.

Figure 6

Figure 6: There exists a critical window of intervention: intervention timing sweep shows a U-shaped dependency, with fastest grokking obtained by activating GeomDR during an early but not initial phase of training.

The cited analyses support a geometric interpretation: GeomDR impedes the optimization of memorizing, spectrally diffuse solutions, restricting the search to compressed solutions wherein generalization is more likely. The geometric constraints imposed by GeomDR correspond to tight upper bounds on the effective number of significant covariance directions, strict low-rank approximation bounds on the covariance matrix, and exponential suppression of representation volume in non-leading directions (see Appendix, Spectral Properties). Fundamentally, the space of representational solutions is re-shaped such that the optimizer is biased toward structured, generalizing representations.

Generalization Across Tasks and Architectures

GeomDR provides substantial acceleration of grokking for both MLPs and Transformers in all tested settings. For MLPs on modular addition, division, and permutation composition, the reduction in time to generalization can exceed an order of magnitude (e.g., 52.5×52.5\times acceleration for modular addition). The effect is more modest but still significant for Transformers, which grok faster even without intervention due to stronger architectural biases. Figure 7

Figure 7

Figure 7

Figure 7

Figure 7

Figure 7

Figure 7: GeomDR's effects generalize across additional synthetic tasks (modular division, permutation composition) in MLPs, and are robust to the intervention schedule, target dimensionality, and start time.

Scalability ablations demonstrate that accurate covariance estimation for GeomDR does not require use of the full dataset; random subsampling (even at 5–25%) maintains acceleration, facilitating extension to larger domains. Comparison with other grokking interventions (weight decay, GrokFast) shows that GeomDR yields superior acceleration and reliability in most synthetic algorithmic settings.

Practical and Theoretical Implications

Empirically, the results establish that effective dimensionality is not just a diagnostic but a manipulable variable, providing a practical handle for controlling the transition from memorization to generalization in synthetic tasks. Architectures benefiting from GeomDR are not confined to MLPs; Transformers show similar, if architecture- and width-dependent, trends.

Theoretically, the findings motivate further investigation into the mechanistic role of geometric invariants in deep learning. The results suggest that transitions in generalization are tightly coupled to measurable and controllable geometric features and that direct manipulation of representation geometry provides a valuable experimental approach for studying learning dynamics. The causal role of representation collapse in grokking demands rigorous modeling in future work, including the analysis of large-scale and real-world domains and the investigation of interaction effects with other forms of explicit and implicit regularization.

Conclusion

This work demonstrates that direct geometric interventions on hidden representations via GeomDR can reliably and dramatically accelerate (and in some configurations suppress) the grokking transition in neural networks across diverse tasks and architectures. Effective dimensionality collapse is an early and manipulable signal for the onset of generalization. The identification of representation geometry as a control signal expands the experimental toolkit for probing the mechanisms of generalization in deep learning and invites the development of new regularization techniques founded on geometric rather than merely parameter-space constraints.


Reference:

"How to Tame Grokking: Representation Geometry as a Control Signal" (2607.11666)

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 3 likes about this paper.