Understanding Deep Neural Networks with Rectified Linear Units (1611.01491v6)

Abstract: In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to global optimality with runtime polynomial in the data size albeit exponential in the input dimension. Further, we improve on the known lower bounds on size (from exponential to super exponential) for approximating a ReLU deep net function by a shallower ReLU net. Our gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature. An example consequence of our gap theorems is the following: for every natural number $k$ there exists a function representable by a ReLU DNN with $k^2$ hidden layers and total size $k^3$, such that any ReLU DNN with at most $k$ hidden layers will require at least $\frac{1}{2}k^{k+1}-1$ total nodes. Finally, for the family of $\mathbb{R}^n\to \mathbb{R}$ DNNs with ReLU activations, we show a new lowerbound on the number of affine pieces, which is larger than previous constructions in certain regimes of the network architecture and most distinctively our lowerbound is demonstrated by an explicit construction of a smoothly parameterized family of functions attaining this scaling. Our construction utilizes the theory of zonotopes from polyhedral theory.

• The paper introduces a polynomial-time algorithm that trains single hidden layer ReLU networks to global optimality, despite exponential dependence on input dimension.
• The paper demonstrates that any continuous piecewise linear function can be represented by a ReLU network, setting bounds on the required depth and size.
• The paper improves lower bounds for approximating ReLU functions with shallower networks, highlighting the critical role of network depth in capturing complexity.

Understanding Deep Neural Networks with Rectified Linear Units

This paper provides an analytical examination of deep neural networks (DNNs) that use Rectified Linear Units (ReLU) as activation functions. The authors explore the family of functions these networks can represent and advance both theoretical understanding and practical applications.

Key Contributions

1. Training to Global Optimality: An algorithm is introduced to train a single hidden layer ReLU DNN to global optimality. The algorithm runs in polynomial time relative to the data size but is exponential in the input dimension. This result is significant as it connects to the broader challenge of efficiently training DNNs while achieving optimal solutions.
2. Expressive Power of ReLU DNNs: The paper presents detailed insights into the expressive power of ReLU DNNs. It provides a proof that any continuous piecewise linear (PWL) function can be represented by a ReLU DNN, with specific bounds on the depth and size required for representation. The authors establish that every function in $L^q(\mathbb{R}^n)$ can be approximated by such networks.
3. Lower Bounds on Network Complexity: The paper improves lower bounds for approximating ReLU deep net functions using shallower networks. These bounds are super-exponential, better defining the complexity involved in simplifying DNN architectures without losing representational capacity.
4. Structured Family for Hard Functions: A family of ReLU network functions is demonstrated where reducing layers significantly increases network size, concretely illustrating the benefit of depth. This family is parametrized smoothly, contrasting the countable sets used in previous complexity theory results.
5. Number of Affine Pieces: The authors explore the number of affine pieces describable by a ReLU network given the network's architecture. A method utilizing zonotopes from polyhedral theory provides a construction demonstrating a new lower bound on the number of affinely linear components a ReLU network can represent. The authors discuss these implications, providing a rigorous argument for the scaling properties of such constructions.

Implications and Future Directions

The implications of this research are profound for both theory and practical implementations in AI:

• Practical Applications: Understanding the precise family of functions representable by ReLU networks informs decisions during model design, impacting performance across applications in classification and more complex machine learning tasks such as image recognition and natural language processing.
• Theoretical Insights: The results contribute to the field of computational complexity regarding neural networks and suggest new avenues for exploring the inherent trade-offs in network design--specifically depth versus width and the accompanying computational demands.
• Future Prospects: The research opens several avenues for further exploration, particularly in extending the results to networks with multiple layers, exploring unsupervised pre-training effects, or considering different non-linear activation functions. Additionally, developing training algorithms with global guarantees for deeper architectures remains a challenge and potential breakthrough area.

This research places significant emphasis on depth as a critical component of DNN architectures' success. While the paper does not sensationalize its findings, it makes substantial strides in understanding the depth-versus-width trade-off in neural networks and enforces depth's role in effectively abstracting hierarchical features from data.