Mathematical framework for feature emergence linked to gradient dynamics

Establish a mathematical framework that precisely characterizes which features emerge during training, how they emerge, and under which conditions they do so, while remaining closely connected to the gradient dynamics of learning for models trained on complex structured inputs.

Background

The paper studies grokking—delayed generalization—in neural networks and seeks a first-principles explanation of how features emerge during training. Despite extensive empirical study of grokking, the authors note that a rigorous framework connecting feature emergence to the gradient dynamics, especially for structured inputs (e.g., group arithmetic), has been lacking.

This open problem motivates the development of the Li framework proposed in the paper, which partitions learning dynamics into lazy, independent, and interactive stages and introduces an energy function governing feature emergence. The problem, as stated, asks for a general mathematical characterization that remains tightly aligned with gradient dynamics.

References

While the phenomenon of grokking, i.e., delayed generalization, has been studied extensively, it remains an open problem whether there is a mathematical framework that characterizes what kind of features will emerge, how and in which conditions it happens, and is still closely connected with the gradient dynamics of the training, for complex structured inputs.

— Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking  (2509.21519 - Tian, 25 Sep 2025) in Abstract

We hypothesize that input distributions shape both the internal representations models learn and the gradients that refine them. How these effects govern the selection and refinement of coset states remains open.

— Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking  (2609.29951 - Zhang et al., 24 Sep 2026) in Section 6, Conclusion (\S\ref{sec:discussion})