There Will Be a Scientific Theory of Deep Learning

Abstract: In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

• The paper introduces a predictive framework of 'learning mechanics' that quantifies gradient dynamics, feature formation, and generalization in deep neural networks.
• It leverages analytically tractable models and asymptotic limits to uncover simple empirical laws governing loss decay, hyperparameter transfer, and neural scaling.
• The paper highlights the universality of inductive biases across architectures, suggesting a shared underlying mechanism for diverse deep learning systems.

Towards a Mechanics of Deep Learning: Evidence, Structure, and Implications

Introduction

The paper "There Will Be a Scientific Theory of Deep Learning" (2604.21691) posits that a rigorous, predictive, and quantitative theory—termed "learning mechanics"—is actively emerging within the deep learning community. Drawing analogies with physics, the authors assert that, rather than remaining a collection of empirical recipes and qualitative observations, deep learning is beginning to yield to systematic theoretical tools capable of predicting macroscopic statistics of optimization, feature formation, and generalization in highly overparameterized, nonlinear systems. The central claim is that a first-principles mechanics of gradient-based learning is feasible, and that existing phenomena across mathematical tractability, scaling limits, empirical laws, hyperparameter disentanglement, and universality constitute convergent lines of evidence supporting this outlook.

Lines of Evidence: Toward a Unified Theory

The argument for learning mechanics is structured around five pillars:

1. Analytically Solvable Systems: Theoretical studies have isolated settings where the gradient-based training of deep networks admits exact or highly tractable solutions. Deep linear networks and kernel methods provide paradigmatic examples: the former enable closed-form characterizations of phase transitions and dynamical phenomena (Figure 1), while the latter, via the neural tangent kernel (NTK), capture learning curves and feature alignment under infinite width and lazy training.

Figure 1: Linearization in the data allows for analytic solutions that match the empirical training curves and the emergence of singular mode learning.

1. Asymptotic and Simplifying Limits: Scaling width, depth, and batch size reveals regularities and limiting behaviors, analogous to mean-field and thermodynamic analyses in physics. Notably, the infinite width limit delineates the dichotomy between lazy (NTK, feature-frozen) and rich (feature-learning) regimes, with crucial dependence on parameterization and output multipliers (Figure 2). Such analysis not only yields insight into the structure of feature learning but also provides a lever for practically robust hyperparameter transfer.

   Figure 2: Modulating the output multiplier induces transitions between lazy and rich training, with the latter facilitating substantial feature adaptation during optimization.

2. Simple Empirical Laws: Macroscopic observables such as test loss, scaling exponents, and sharpness often obey concise power-law relationships across parameter count, data size, and compute (Figure 3). For instance, empirical neural scaling laws demonstrate predictable performance improvements as scale increases. Furthermore, training dynamics are frequently poised near the edge of stability, with aggregate statistics like Hessian sharpness stabilizing at predictable values set by the learning rate (Figure 4).

   Figure 3: The loss of large neural networks decays according to predictable neural scaling laws, forming robust power-law relations with dataset size, compute, and model parameters.

   

   Figure 4: Gradient descent typically operates at the edge of stability, with Hessian sharpness tightly controlled by the learning rate in full-batch settings.

3. Hyperparameter Disentanglement: Careful theoretical analysis shows that architectural and optimization hyperparameters can be systematically decoupled, particularly under correct parameterization and normalization strategies. Tensor Programs and $\mu$P parameterization, for example, facilitate the transferability of optimized learning rates across different widths, enhancing both practical model development and theoretical tractability (Figure 5).

   Figure 5: The $\mu$P parameterization enables the learning rate for optimally-trained networks to remain invariant with model width, in contrast to standard parameterization, aiding efficient scaling and hyperparameter transfer.

4. Universality Across Systems and Data: Across architectures, modalities, and optimization schemes, deep networks exhibit analogous inductive biases and convergent representations, indicating that the effective macroscopic variables are shared across design details (Figure 6). As exemplified by the convergence in learned distributions of diffusion architectures and the increasing similarity between vision and language representations in larger models, the theory points to an underlying universality class for feature formation and inductive bias.

Figure 6: Universality is observed across diverse architectures and data modalities, with learned representations and outputs converging despite architectural differences.

Structure and Scope of Learning Mechanics

The paper advocates for "learning mechanics" to be mathematical, predictive, and operationally useful, while remaining explicit about its limits of applicability. Seven desiderata are outlined for a mature mechanics:

• Foundation in first-principles mathematics
• Explicit quantitative predictions
• Breadth across training, representations, and generalization
• Intuitive and simple explanatory constructs
• Practical utility for architecture and optimization design
• Humility about domains of validity and breakdown

Importantly, this mechanics is not intended as an exhaustive microscopic description but instead as a model for aggregate, experimentally falsifiable predictions.

Relationships to Other Theoretical Paradigms

The authors discuss synergies and complementary roles for statistical learning theory, information-theoretic approaches, and mechanistic interpretability. While classical PAC/VC theory emphasizes worst-case generalization and expressivity, and information theory foregrounds compression and minimal description length, learning mechanics focuses on the concrete, average-case dynamics of gradient flow and stochastic descent. Mechanistic interpretability, compared to a biological dissection, is envisioned as symbiotic with the physics-inspired learning mechanics, with the latter providing the mathematical structure upon which formal interpretability may eventually rest.

Addressing Skepticism and Open Directions

The paper systematically addresses classical objections: the absence of theory is not evidence of impossibility, local theories (e.g., for scaling laws and optimizer hyperparameters) are already impactful, and the field's rapid empirical progress supplies new phenomena ripe for theoretical systematization. Limits are acknowledged, with the theory explicitly aiming for validity in the aggregate and being forthright about breakdowns outside its applicable regime.

The open questions articulated span several axes: the search for nonlinear, analytically tractable models capturing key phenomena not explainable by linearization; identifying the minimal sufficient statistics of data that drive generalization; formulating precise notions of learned features; predicting scaling law exponents from first principles; and eliminating the need for manual hyperparameter tuning through deeper understanding.

Implications and Future Trajectories

The development of a predictive, physically inspired theory of deep learning has multifaceted implications. Practically, it is anticipated to yield tools for principled scaling, hyperparameter elimination, sensitivity analysis, dataset design, and robust model attribution. Theoretically, such a mechanics could illuminate foundational questions in out-of-distribution generalization, algorithmic alignment, and the convergence of learned representations across architectures and modalities. The interaction with mechanistic interpretability is likely to nourish both fields, with interpretability supplying fine-grained empirical targets and learning mechanics offering abstract, transferable invariants.

The empirical successes of neural scaling laws, hyperparameter transfer, and convergence properties in dynamic regimes such as the edge of stability constitute strong numerical evidence for the emergence of learning mechanics. The bold claim supported by the evidence is that the nonlinear interaction of architecture, data statistics, and stochastic optimization can and will be quantitatively characterized—contradicting the view that a unifying scientific theory will remain out of reach.

Conclusion

The synthesis in "There Will Be a Scientific Theory of Deep Learning" (2604.21691) delineates concrete foundations, testable regularities, and research priorities for the mechanics of deep learning. By focusing on aggregate, emergent, and universal behaviors, the proposed theory aspires to systematize the empirical and phenomenological progress that has defined deep learning over the past decade. Emerging theoretical regularities, mathematical tools, and empirical laws collectively signal that deep learning is increasingly amenable to physics-style theory, with implications across AI development, theoretical neuroscience, safety, and interpretability. The trajectory outlined predicts not only maturation in understanding neural architectures but also enhanced rigor, safety, and innovation in next-generation AI systems.