Configuration-independent norm-compression threshold

Establish whether the memorization-to-generalization transition occurs when the ratio of the current weight norm to its value at memorization crosses a configuration-independent threshold in the range 0.3–0.5, and determine whether the time to reach that threshold follows the reported scaling law.

Background

The paper observes monotonic weight-norm compression between memorization and generalization in 14 configurations with nontrivial grokking gaps. The median ratio of the norm at generalization onset to the norm at memorization is 0.42, with an interquartile range of 0.31–0.54.

The authors conjecture that this ratio may define a regime-transition threshold, while noting that the available sample is modest. Verification would clarify whether norm compression is merely correlated with grokking or supplies a configuration-independent criterion for predicting the transition.

References

The memorization-to-generalization transition occurs when the weight norm ratio $\lVert\theta(t)\rVert / \lVert\theta(T_{\mathrm{mem})\rVert$ crosses a threshold $r*$ in the range $0.3$--$0.5$. If $r*$ is approximately configuration-independent, then the time to reach it is governed by the scaling law eq:exponents, with weight decay $\lambda$ controlling the compression rate and data fraction $D$ controlling how much compression is needed.

— Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking  (2609.10657 - Kataria, 9 Sep 2026) in Conjecture 2 ("Norm compression threshold"), Section 6