The Computational Advantage of Depth: Learning High-Dimensional Hierarchical Functions with Gradient Descent (2502.13961v3)

Abstract: Understanding the advantages of deep neural networks trained by gradient descent (GD) compared to shallow models remains an open theoretical challenge. In this paper, we introduce a class of target functions (single and multi-index Gaussian hierarchical targets) that incorporate a hierarchy of latent subspace dimensionalities. This framework enables us to analytically study the learning dynamics and generalization performance of deep networks compared to shallow ones in the high-dimensional limit. Specifically, our main theorem shows that feature learning with GD successively reduces the effective dimensionality, transforming a high-dimensional problem into a sequence of lower-dimensional ones. This enables learning the target function with drastically less samples than with shallow networks. While the results are proven in a controlled training setting, we also discuss more common training procedures and argue that they learn through the same mechanisms.

• The paper introduces hierarchical target functions (SIGHT and MIGHT) to reveal how depth reduces sample complexity in high-dimensional learning.
• The analysis shows that gradient descent enables progressive dimensionality reduction, outperforming shallow networks in feature extraction.
• Numerical simulations confirm that deep architectures leverage hierarchical structures, offering practical insights for efficient model design.

Analyzing the Computational Advantage of Depth in Learning High-Dimensional Hierarchical Functions

This paper presents a compelling theoretical investigation into the computational advantages of deep neural networks over shallow models when trained using gradient descent (GD). The authors introduce hierarchical features within target functions, including both single and multi-index Gaussian hierarchical targets (SIGHT and MIGHT), to examine how depth enables more efficient learning through reduced sample complexity and enhanced feature learning. The exploration centers on how depth fundamentally influences the learning dynamics and enables high-dimensional problems to be transformed into sequences of lower-dimensional ones.

Key Contributions

1. Theoretical Framework for Hierarchical Learning: The paper introduces MIGHT and SIGHT functions that incorporate varying latent subspace dimensionalities. This hierarchical structure is used to paper deep networks analytically, demonstrating how GD-trained networks can exploit these structures more effectively than shallow networks.
2. Analytical Insights and Theorems: The authors rigorously prove that feature learning with GD in deep networks results in a series of dimensionality reductions. For example, in the learning of a specific SIGHT function with a three-layer neural network, they show that the network first recovers intrinsic feature structures utilizing $\tilde{O}(d^{\epsilon_1 + 1})$ samples, subsequently reconstructs a non-linear feature mapping with $\tilde{O}(d^{k \epsilon_1})$ samples, and finally fits the target function using $\tilde{O}(1)$ samples. This is a significant reduction in sample complexity compared to shallow networks.
3. Implications for Network Depth: The findings substantiate that the computational advantage of depth in neural networks arises from this multidimensional hierarchy reduction capability, which is unique to deeper architectures. This "coarse-graining" mechanism allows networks to distill information progressively, mirroring methods such as renormalization in physics.
4. Numerical Simulations and Practical Implications: The paper includes numerical simulations that substantiate the theoretical findings, demonstrating that standard training methodologies, including backpropagation, can also leverage these hierarchical structures to significant effect. This illustrates the practical utility of the proposed models beyond idealized training scenarios.
5. Discussion on Generalization to Deeper Networks: It considers extensions beyond the three-layer networks, providing preliminary analyses for MIGHT functions and hinting at the broader applicability of these insights to even deeper neural architectures.

Implications and Future Directions

• Practical Relevance: The reduction in effective dimensionality suggests pathways for designing more efficient deep learning architectures in practice, where feature hierarchies in data can be more fully exploited.
• Theoretical Advancements: The paper advances the theoretical understanding of how depth contributes to non-linear function approximation and learning, paving the way for quantifiable improvements in network design.
• Addressing Complex Targets: Future research could extend these findings to more complex real-world datasets, where hierarchical features are more pronounced, validating these theoretical insights in broader practical settings.
• Extending Beyond Gaussian Assumptions: While the Gaussian setting offers analytical tractability, future work might consider other data distributions to broaden the applicability of these results.

In summary, this paper offers a deep exploration of the advantages of depth in neural networks, both theoretically and numerically, with substantial potential implications for the development of future AI models and algorithms. The paradigm introduced by hierarchical target functions and their subsequent learning dynamics could significantly impact how deep learning models are understood and improved.