On the Expressive Power of Deep Neural Networks (1606.05336v6)

Abstract: We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.

• The paper demonstrates that network expressivity grows exponentially with depth through novel trajectory length measures.
• It reveals the critical role of initial layers by showing heightened sensitivity to input perturbations affecting performance.
• Additionally, the study proposes trajectory regularization as an efficient method to stabilize training similar to batch normalization.

An Analysis of the Expressive Power of Deep Neural Networks

The paper, "On the Expressive Power of Deep Neural Networks," presents a rigorous examination of how structural properties of neural networks affect their capacity to compute functions. The research seeks to quantify the relationship between architectural aspects, such as depth and width, and the expressive power of neural networks. The authors introduce novel measures of expressivity, unified under the concept of trajectory length, to address fundamental questions surrounding the complexity and behavior of neural networks. This essay will provide an expert analysis of the key findings, implications, and future directions discussed in the paper.

Key Findings

1. Exponential Growth of Complexity with Depth: The paper introduces measures of expressivity that quantify the non-linearity of the functions computed by neural networks. These measures demonstrate that the complexity grows exponentially with the depth of the network. This is evident from the exponential increase in trajectory length, which tracks how the network’s output changes as the input sweeps along a one-dimensional path.
2. Importance of Initial Layers: The findings show that networks are more sensitive to perturbations in their lower layers, indicating that optimizing weights in initial layers has a significant impact on overall performance. The network's robustness to noise and overall performance is compromised by noisy weights in lower layers.
3. Trajectory Regularization and Batch Normalization: The paper highlights that trajectory regularization, a proposed method inspired by the effect of batch normalization (Bn), stabilizes the representation learned by the network similar to Bn but is computationally more efficient. This stabilization effect is attributed to controlling the growth of trajectory length during training.

Implications and Theoretical Contributions

Upper and Lower Bounds on Expressivity:

The authors derive tight, provable upper bounds on the number of activation patterns a neural network can realize given its architecture, using ideas from hyperplane arrangements in computational geometry. These bounds match the lower bounds produced by previous work, illustrating that deep networks inherently exhibit a higher number of linear regions compared to shallow networks with equivalent parameters. The bounds apply universally across different weight settings, providing a comprehensive framework for comparing the expressive power across network architectures.

Role of Activation Patterns and Neuron Transitions:

The paper formalizes the concept of activation patterns and neuron transitions as measures of expressivity. Activation patterns give insight into how networks partition input space into regions where each region computes a distinct linear function. The exponential growth in trajectory length with network depth is shown to correlate strongly with the number of activation pattern transitions, underscoring the intricate structure and behavior of deep networks.

Insights into Network Training and Regularization:

The empirical results demonstrating the exponential growth of trajectory length with depth offer insights into training dynamics. Networks tend to increase their trajectory length during training, enhancing expressivity yet risking instability. Techniques such as batch normalization and trajectory regularization help manage this growth, striking a balance between expressivity and stability, which is critical for robust learning.

Future Developments in AI

The findings raise several intriguing directions for future research:

• Adversarial Robustness: Given the relationship between trajectory length and network sensitivity, understanding how trajectory growth contributes to adversarial vulnerabilities can drive the development of more robust models.
• Optimization Landscapes: The conception of input space partitioning into convex polytopes can inform optimization techniques, potentially improving convergence rates and training efficiency for deep networks.
• Regularization Techniques: Extending trajectory regularization to more sophisticated forms could yield better generalization performance without the computational overhead associated with batch normalization.

Conclusion:

The paper provides a detailed theoretical framework for understanding the expressive capabilities of deep neural networks, highlighted by the novel use of trajectory length. The insights gained from these findings hold significant implications for network design, training methodologies, and future research in machine learning. By firmly establishing the exponential depth dependence in expressivity and the critical importance of lower layers, this work contributes to the deeper understanding of neural network architectures’ profound complexity and potential.