Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity (1602.05897v2)

Abstract: We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.

• The paper demonstrates that random initialization creates expressive representations, allowing approximation of any function in the hypothesis space using last-layer tuning.
• The authors establish a duality between neural networks and compositional kernels, highlighting new pathways for network architecture design.
• They recommend employing convolutional structures and ReLU activations to ensure robustness even as network depth increases and risks of representation collapse emerge.

An Analytical Examination of Neural Network Initialization and Kernel Duality

This paper, entitled "Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity," presents an exploration of the connections between neural networks (NNs) and compositional kernels, aimed at elucidating several critical but less understood aspects of deep learning. The authors develop a theoretical framework that leverages the concept of dual spaces to clarify the role of random initialization in neural networks and propose principles for network architecture design, particularly advocating for both convolutional network structures and initialization strategies grounded in compositional kernel dualities.

Key Insights and Results

The primary contribution of this work is establishing a theoretical duality between neural network architectures and compositional kernels. Several implications arise from this dual view:

1. Role of Initialization: The paper demonstrates that common random initialization schemes generate sufficiently expressive representations such that all functions within a given hypothesis space can be approximated by merely tuning the weights of the last layer. This insight is both theoretically significant and practical, providing a plausible explanation for the observed success of training algorithms despite the prevalence of non-convex optimization problems and numerous local minima in NN training.
2. Kernel and Network Duality: The authors bridge neural networks and kernel methods by defining a kernel space corresponding to every NN architecture. This connection potentially unveils new design strategies for architectures aimed at particular tasks, supporting adherence to existing practices and stimulating new insights.
3. Architectural Recommendations: Several specific architectural implications are derived from the theory:
   • Convolutional networks are better fitted for visual and acoustic tasks compared to fully connected architectures.
   • The ReLU activation function (max(x,0)) is highlighted for its favorable properties and robustness to initialization variations due to its positive homogeneity.
   • By adjusting activation functions, some layers can be omitted without loss of network expressivity.
4. Depth and Representation Collapse: The paper further claims that as the number of layers in a fully connected network increases without bounds, the dual kernel converges to a degenerate form, articulating practical limits on network depth when using non-linear activations.

Implications for Neural Network Design

The duality framework proposed in this paper provides several valuable implications for designing neural networks that are theoretically informed beyond empirical heuristics. The paper sheds light on why certain empirical practices, such as weight initialization and the use of specific activation functions like ReLU, contribute to the success of neural network training. Additionally, it suggests using compositional kernels as guidelines for deducing which functions are feasible to learn, enhancing our understanding of learnable classes with specific architectures.

The authors hypothesize that the robustness of ReLU activations and the proposed initialization scheme contribute to their widespread use in practice. This resilience stems from ReLU's positive homogeneity, which maintains expressivity even when the scale of initial weights is altered. The initialization method outlined aligns closely with common practice, but with subtle adjustments that may improve efficacy.

Future Directions

While this paper offers strong theoretical foundations, it opens avenues for further investigation. Subsequent research could provide refined bounds on expressivity or convergence, perhaps leading to improvements in the polynomial dependencies on network depth and other parameters. Moreover, expanding the framework to include more intricate network operations like max-pooling or recursive components presents promising directions. Lastly, the cross-pollination of ideas between random features and neural network embeddings offers a fruitful domain to explore.

In summary, this paper presents a robust framework for understanding neural networks through the lens of compositional kernel spaces, providing valuable insights into initialization, activation function selection, and architectural design strategies. This dual approach enriches both theoretical analysis and practical implementations, paving the way for more informed NN architecture design and initialization practices.