Mathematical Foundations of Feature Learning in Deep Neural Networks

Establish a rigorous mathematical foundation that explains feature learning in deep neural networks, precisely characterizing how trained models extract and encode information from high-dimensional inputs and identifying the mechanisms and implicit biases that govern this process.

Background

Feature learning refers to the process by which deep neural networks extract informative representations from high-dimensional inputs. Despite extensive empirical success, a principled mathematical understanding of feature learning—its drivers, implicit biases, and the structures it induces—remains incomplete.

This paper studies the Neural Feature Ansatz (NFA) and related low-rank phenomena as possible mechanisms underlying feature learning, proving exact and asymptotic results for deep linear networks and exploring limitations for nonlinear architectures. These results highlight depth-dependent behavior and the role of initialization and weight decay, but a general theoretical framework for feature learning itself remains to be developed.