- The paper demonstrates that inducing a rapid collapse in representation dimensionality via GeomDR is key to accelerating the grokking transition in neural networks.
- It details how spectral regularization manipulates hidden representations by constraining covariance spectra, leading to improved generalization across tasks.
- The work provides empirical evidence from MLPs and Transformers, with quantitative benchmarks highlighting the practical benefits of geometric intervention.
Representation Geometry as a Control Signal for Grokking
Introduction
This work presents a comprehensive study of the relationship between representation geometry—specifically, representation dimensionality—and the phenomenon of grokking in neural networks. Grokking refers to the empirical observation that, in certain algorithmic tasks, neural networks first memorize the training data but only acquire strong generalization capabilities after a prolonged training regime. It has remained unclear whether the delayed onset of generalization is merely correlated with, or mechanistically driven by, changes in the geometry of learned representations. The paper introduces Geometric Dimensionality Regularization (GeomDR), a spectral regularizer targeting the covariance spectra of hidden representations, and demonstrates that manipulating representation geometry provides direct control over the timing and character of grokking across various architectures and synthetic algorithmic tasks.
Grokking and Representation Geometry
Baseline experiments replicate delayed generalization in synthetic modular arithmetic tasks. Trajectories of effective dimensionality and local neighborhood distance through training reveal sharp reorganization phases in the representation space coinciding with the test accuracy transition, particularly in deeper layers. In all cases, dimensionality collapse—a reduction of the effective number of active covariance dimensions—precedes the onset of generalization, suggesting a tight temporal relationship between these geometric signatures and functional grokking.





Figure 1: Baseline learning: effective dimensionality, local neighborhood metric, and accuracy curves, illustrating abrupt geometry transitions immediately prior to test accuracy increase.
Unsmoothed geometric traces show that this is not a slow, continuous collapse; rather, a rapid, punctuated change in the representation metric signals the imminent success in test accuracy. These metrics jointly support the hypothesis that a reduction in the complexity of the internal representation is a necessary precursor for generalization on these tasks.
Geometric Dimensionality Regularization (GeomDR)
GeomDR is a representation-level objective that penalizes covariance outside a user-specified number of dominant principal components in each hidden layer. For a target dimensionality d∗, the spectral regularizer LGeomDR(ℓ)=j=d∗+1∑dμj(ℓ) suppresses variance in trailing components. The regularizer is ramped in gradually after an initial unconstrained phase, in order to not disrupt the phase of model fitting dynamics when memorizing solutions first emerge. The full loss is L=Ltask+λ(t)LGeomDR.
The regularization schedule and the choice of d∗ critically affect the model's learning trajectory. Sweeps over these parameters show robust acceleration in grokking, provided the regularization is neither too strong nor too weak and the target dimensionality is not overly restrictive.





Figure 2: Imposing target dimensionalities via GeomDR leads to immediate and pronounced geometric transitions and substantially accelerated grokking.
Intermediate values for both regularization strength λmax and target dimension d∗ (16≤d∗≤64 for modular addition) yield consistent and large effects, while over-compression (d∗≤8) can destabilize convergence. Temporal sweeps demonstrate that GeomDR is most effective when applied after an initial period of memorization but before the transition to rigid, task-agnostic representations.
Figure 3: Schedule sweep: intermediate ramp and regularization yield maximal acceleration of grokking; overly aggressive settings can suppress generalization entirely.
Geometric Causality and Mechanistic Implications
Systematic ablations establish that geometric intervention via GeomDR provides causal leverage over the grokking transition. Across all tasks and architectures, collapse in effective dimensionality tightly precedes, and with GeomDR almost coincides with, the transition to test generalization. For MLPs, well-chosen regularizer configurations compress the lag between collapse and grokking from hundreds of thousands to mere thousands of update steps.
Figure 4: Intermediate and large d∗ values accelerate grokking; aggressive compression slows and destabilizes learning.
Figure 5: Representation collapse consistently precedes the grokking step; GeomDR narrows the collapse–grokking lag.
Figure 6: There exists a critical window of intervention: intervention timing sweep shows a U-shaped dependency, with fastest grokking obtained by activating GeomDR during an early but not initial phase of training.
The cited analyses support a geometric interpretation: GeomDR impedes the optimization of memorizing, spectrally diffuse solutions, restricting the search to compressed solutions wherein generalization is more likely. The geometric constraints imposed by GeomDR correspond to tight upper bounds on the effective number of significant covariance directions, strict low-rank approximation bounds on the covariance matrix, and exponential suppression of representation volume in non-leading directions (see Appendix, Spectral Properties). Fundamentally, the space of representational solutions is re-shaped such that the optimizer is biased toward structured, generalizing representations.
Generalization Across Tasks and Architectures
GeomDR provides substantial acceleration of grokking for both MLPs and Transformers in all tested settings. For MLPs on modular addition, division, and permutation composition, the reduction in time to generalization can exceed an order of magnitude (e.g., 52.5× acceleration for modular addition). The effect is more modest but still significant for Transformers, which grok faster even without intervention due to stronger architectural biases.





Figure 7: GeomDR's effects generalize across additional synthetic tasks (modular division, permutation composition) in MLPs, and are robust to the intervention schedule, target dimensionality, and start time.
Scalability ablations demonstrate that accurate covariance estimation for GeomDR does not require use of the full dataset; random subsampling (even at 5–25%) maintains acceleration, facilitating extension to larger domains. Comparison with other grokking interventions (weight decay, GrokFast) shows that GeomDR yields superior acceleration and reliability in most synthetic algorithmic settings.
Practical and Theoretical Implications
Empirically, the results establish that effective dimensionality is not just a diagnostic but a manipulable variable, providing a practical handle for controlling the transition from memorization to generalization in synthetic tasks. Architectures benefiting from GeomDR are not confined to MLPs; Transformers show similar, if architecture- and width-dependent, trends.
Theoretically, the findings motivate further investigation into the mechanistic role of geometric invariants in deep learning. The results suggest that transitions in generalization are tightly coupled to measurable and controllable geometric features and that direct manipulation of representation geometry provides a valuable experimental approach for studying learning dynamics. The causal role of representation collapse in grokking demands rigorous modeling in future work, including the analysis of large-scale and real-world domains and the investigation of interaction effects with other forms of explicit and implicit regularization.
Conclusion
This work demonstrates that direct geometric interventions on hidden representations via GeomDR can reliably and dramatically accelerate (and in some configurations suppress) the grokking transition in neural networks across diverse tasks and architectures. Effective dimensionality collapse is an early and manipulable signal for the onset of generalization. The identification of representation geometry as a control signal expands the experimental toolkit for probing the mechanisms of generalization in deep learning and invites the development of new regularization techniques founded on geometric rather than merely parameter-space constraints.
Reference:
"How to Tame Grokking: Representation Geometry as a Control Signal" (2607.11666)