Mathematical Foundations of Feature Learning in Deep Neural Networks

Establish a rigorous mathematical foundation that explains feature learning in deep neural networks, precisely characterizing how trained models extract and encode information from high-dimensional inputs and identifying the mechanisms and implicit biases that govern this process.

Background

Feature learning refers to the process by which deep neural networks extract informative representations from high-dimensional inputs. Despite extensive empirical success, a principled mathematical understanding of feature learning—its drivers, implicit biases, and the structures it induces—remains incomplete.

This paper studies the Neural Feature Ansatz (NFA) and related low-rank phenomena as possible mechanisms underlying feature learning, proving exact and asymptotic results for deep linear networks and exploring limitations for nonlinear architectures. These results highlight depth-dependent behavior and the role of initialization and weight decay, but a general theoretical framework for feature learning itself remains to be developed.

References

Understanding feature learning is an important open question in establishing a mathematical foundation for deep neural networks.

— On the Neural Feature Ansatz for Deep Neural Networks  (2510.15563 - Tansley et al., 17 Oct 2025) in Abstract (page 1)

Several directions remain open. Our analysis focuses on one-layer linear attention, spherical-harmonic features, and an idealized two-stage training procedure in which the scalar readout is learned from sufficient additional data. It would be useful to understand whether the same separation persists for random features or learned embeddings, jointly trained readouts, softmax attention, and deeper Transformer architectures.

— In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners  (2610.01712 - Gu et al., 1 Oct 2026) in Section “Conclusion and Future Work”

We conjecture that a clear decomposition of vision transformers into features (final activations) and weight (final layer) does not hold (though it does for language transformers).

— Why Heavy-Tailed Weight Spectra Predict Model Quality  (2610.13123 - Wilson et al., 9 Oct 2026) in Appendix, Section "When last-layer free energy does not predict model quality"