Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (2402.05626v6)

Abstract: In this paper, we study the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss using gradient descent (GD). We identify the stationary points of such networks, which significantly slow down loss decrease during training. To capture such points while accounting for the non-differentiability of the loss, the stationary points that we study are directional stationary points, rather than other notions like Clarke stationary points. We show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, it must be a local minimum. Moreover, for the scalar-output case, the presence of an escape neuron guarantees that the stationary point is not a local minimum. Our results refine the description of the saddle-to-saddle training process starting from infinitesimally small (vanishing) initialization for shallow ReLU-like networks: By precluding the saddle escape types that previous works did not rule out, we advance one step closer to a complete picture of the entire dynamics. Moreover, we are also able to fully discuss how network embedding, which is to instantiate a narrower network with a wider network, reshapes the stationary points.

• The paper reveals that stationary points without escape neurons are local minima, clarifying key network behavior.
• The paper demonstrates that the presence of an escape neuron in scalar-output scenarios enables saddle escaping, refining training dynamics under vanishing initialization.
• The paper shows that network embedding via unit replication preserves local minima and reshapes the loss landscape, influencing optimization efficacy.

Delving Into the Loss Landscape of Shallow ReLU-like Neural Networks

Introduction to the Study

The exploration of neural networks often brings researchers into complex territories of understanding how the intricacies of network architecture, activation functions, and training methods impact the overall learning and generalization capabilities of models. A paper examines the loss landscape of shallow networks with ReLU-like activation functions, providing insight into the characteristics of stationary points, the dynamics surrounding saddle point escaping, and the implications of network embedding.

Stationary Points and Loss Landscape Characterization

The research systematically characterizes stationary points in shallow neural networks, integrating ReLU-like activations with the empirical squared loss. The investigation highlights the unique challenges posed by the non-differentiability of ReLU-like functions, prompting a nuanced approach to characterize stationary points. Key findings suggest:

• Stationary points devoid of "escape neurons" are invariably local minima, with escape neurons defined via first-order conditions.
• In scalar-output scenarios, the presence of an escape neuron guarantees a stationary point is not a local minimum, refining the understanding of training dynamics from vanishing initialization.
• The paper also elaborates on how the escape neurons' parameter changes are central to the saddle escaping process, linking it directly to the adjustments within the network's architecture.

Training Dynamics and Initialization Regimes

The training dynamics of shallow networks, especially under vanishing initialization, follow a noticeable saddle-to-saddle pattern. This pattern is characterized by training phases with intermittent steep loss declines followed by plateaus. The presence of small live neurons – associated with escape neurons – underpins these dynamics, revealing how the network gradually acquires complexity by adding more expressive features, akin to fitting more kinks in a piecewise linear function modeled by the network.

Network Embedding and Stationary Points

An innovative aspect of this research is its examination of how network embedding, the process of embedding a narrower network within a larger one, reshapes stationary points. It was found that:

• Embedding a network by unit replication, if done under certain conditions, preserves the local minima, barring the creation of escape neurons.
• The embedding method significantly impacts the optimization landscape, supporting the intuitive notion that over-parameterization can influence the ease of training and the achievement of lower training loss.

Related Works and Theoretical Foundations

The paper situates its findings within the broader discourse on stationary points in neural network optimization, referencing seminal works that have laid the groundwork for understanding how local minima, saddle points, and other critical points shape the loss landscape. This paper goes further by elucidating the role of non-differentiability in sculpting the optimization landscape of networks with ReLU-like activations.

Implications and Speculations on Future Developments

The implications of this paper are manifold, touching on both theoretical insights and practical considerations in neural network training. The characterization of stationary points and the dynamics of saddle escaping enrich the theoretical understanding of why certain training initialization scales and network architectures favor or hinder effective training.

Looking ahead, the refined understanding of loss landscapes and network embedding offers a promising avenue for developing more robust and theoretically grounded training algorithms. There might also be opportunities to extend these insights to more complex network architectures and other types of activation functions.

Conclusion

The investigation into the loss landscape of shallow ReLU-like neural networks uncovers pivotal dynamics that govern training behavior and the realization of local minima. The paper's findings on the role of escape neurons, the significance of network embedding, and the implications for training dynamics from vanishing initialization lay solid ground for future explorations aimed at demystifying the complex interplay between network architecture, loss landscapes, and learning efficacy.