Incorporate Training Dynamics to Explain Double Descent and Grokking

Develop an extension of the concept–text and skill–concept graph framework that explicitly models training time and the number of training epochs, and determine whether this extended framework can reproduce and explain the double descent and grokking phenomena observed in neural network training.

Background

The paper introduces a unified information-theoretic framework based on concept–text and skill–concept bipartite graphs to explain compute-optimal scaling, emergence, and plateaus in LLMs. While the framework captures many scaling phenomena via static graph properties and iterative peeling (analogous to LDPC decoding), it abstracts away training dynamics.

In the conclusion, the authors explicitly note that training time is not modeled, which prevents the framework from addressing dynamic phenomena such as double descent and grokking. They explicitly flag this as an open question and suggest incorporating training epochs or proposing a different framework to address these phenomena.

References

There are some open questions and considerations worth exploring. We do not take training time into account in our framework. Therefore, we do not explain (or attempt to explain) empirical phenomena such as double descent or grokking . Perhaps future work can either incorporate training epochs in our framework or propose a different novel framework to explain them.

— An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models  (2410.01243 - Nayak et al., 2024) in Conclusion, final paragraph

Given the empirical nature of \cref{eq:repetition-law}, we leave the interpretation of this term as driving early stopped model-wise double descent, and the apparent ``optimal-compute-wise" double descent, to future work, and qualitatively draw four scenarios in \cref{fig:gamma_frontiers}.

— How to scale your HEP ML models: A recipe for robust architecture comparisons at scale  (2610.06784 - Vigl et al., 5 Oct 2026) in Appendix, Section “Data repetition: hyperparameter scaling and optimal-compute-wise double descent,” paragraph “Optimal-compute-wise double descent?”

It is possible, if apparently unlikely, that with a much larger training dataset performance would begin to increase again and cross a $99\%$ accuracy threshold, so-called `grokking' .

— Deep Learning of Topological Equivalence Classes  (2610.21981 - Disney-Hogg et al., 15 Oct 2026) in Section 4, Discussion