Error bounds for approximations with deep ReLU networks (1610.01145v3)

Abstract: We study expressive power of shallow and deep neural networks with piece-wise linear activation functions. We establish new rigorous upper and lower bounds for the network complexity in the setting of approximations in Sobolev spaces. In particular, we prove that deep ReLU networks more efficiently approximate smooth functions than shallow networks. In the case of approximations of 1D Lipschitz functions we describe adaptive depth-6 network architectures more efficient than the standard shallow architecture.

• The paper establishes rigorous upper and lower bounds for approximating functions in Sobolev spaces, showing that deep ReLU networks achieve logarithmic complexity.
• It introduces adaptive depth-6 architectures that efficiently approximate one-dimensional Lipschitz functions, enabling efficient squaring and multiplication operations.
• The analysis contrasts deep and shallow networks, revealing that deep networks require significantly lower complexity for approximating smooth functions.

Error Bounds for Approximations with Deep ReLU Networks

The paper "Error bounds for approximations with deep ReLU networks" by Dmitry Yarotsky is a comprehensive investigation into the expressive power of deep and shallow ReLU networks, particularly focusing on their ability to approximate functions within Sobolev spaces.

The primary result of this paper is the establishment of rigorous upper and lower bounds for network complexity when approximating functions in $\mathcal{W}^{n,\infty}$ Sobolev spaces. The central finding demonstrates that deep ReLU networks can approximate smooth functions markedly more efficiently than shallow networks. Additionally, the paper presents novel adaptive depth-6 network architectures which show improved efficiency over standard shallow architectures for one-dimensional Lipschitz functions.

Key Contributions

1. Model and Approximation Theory:
   • A general ReLU network is defined and its approximation capabilities are studied in the context of functions from Sobolev spaces $\mathcal{W}^{n,\infty}([0,1]^d)$.
   • The complexity of networks is measured using conventional metrics: depth, number of weights, and the number of computation units.
2. Upper Bounds:
   • Function Squaring and Multiplication:
     • The paper introduces efficient ReLU network approximations for the squaring function $f(x) = x^2$, showing that it can be approximated with error $\epsilon$ by a network of depth and complexity $O(\ln (1/\epsilon))$.
     • Extending this to multiplication, the network can approximate products of bounded numbers with the same logarithmic complexity, leveraging a practical instance of chaining approximations for basic operations.

- General Smooth Functions: - For functions in $F_{d,n}$ (the unit ball in $\mathcal{W}^{n,\infty}$), the paper establishes a ReLU network architecture of depth $O(\ln (1/\epsilon))$ and complexity $O(\epsilon^{-d/n} \ln (1/\epsilon))$ capable of approximating any function in this space with error $\epsilon$.

- Adaptive Architectures for 1D Lipschitz Functions: - In cases where the network structure can be adapted depending on the function, the complexity can be further reduced. Specifically, for one-dimensional Lipschitz functions, depth-6 ReLU networks can achieve approximations with complexity $O(1/(\epsilon \ln (1/\epsilon)))$.

1. Lower Bounds:
   • Continuous Nonlinear Widths:
     • Under the assumption of continuous model selection, any architecture that approximates functions in $F_{d,n}$ with error $\epsilon$ must have at least $c \epsilon^{-d/n}$ connections and weights.

- VC-Dimension and General Lower Bounds: - For fixed architectures without the continuous selection assumption, the paper utilizes results from VC-dimension theory to establish a lower bound: a network that approximates functions with error $\epsilon$ cannot have fewer than $c \epsilon^{-d/(2n)}$ weights. - If the network depth grows logarithmically with $\epsilon$, the lower bound is tighter: $c \epsilon^{-d/n} \ln^{-2p-1}(1/\epsilon)$ for depth scaling as $O(\ln^p(1/\epsilon))$.

- Adaptive Network Architectures: - There exist functions in $\mathcal{W}^{n,\infty}$ for which the number of units needed for $\epsilon$-approximation is not $o(\epsilon^{-d/(9n)})$, highlighting the limitations even for adaptive architectures.

1. Comparison with Shallow Networks:
   • The presented results strongly indicate that for very smooth functions, deep networks are much more efficient than shallow ones. Specifically, while deep networks can achieve logarithmic complexity for such functions, shallow networks exhibit a polynomial growth in complexity.

Implications and Future Directions

The implications of these findings are significant for both theoretical and practical aspects of neural network design. From a theoretical standpoint, the results underline the advantages of depth in network design, particularly for complex or smooth functions. Deep networks exhibit superior efficiency compared to shallow networks, thus providing a robust theoretical foundation for their successful application in many contemporary tasks.

Practically, the insights into adaptive architectures point towards more efficient network designs that leverage hierarchical and structural characteristics of the data, a common feature in real-world problems. This could lead to more computation and resource-efficient models, particularly crucial for environments with limited computing resources.

Future research directions could explore deeper into other forms of activation functions, the impact of specific data structures, and the development of more sophisticated adaptive network strategies. Additionally, finer-grained analyses of VC-dimensions for deep networks could yield even tighter bounds and more nuanced understanding of network efficiency.

In summary, Yarotsky's paper significantly advances our understanding of the relationship between network depth, complexity, and expressiveness, offering key insights into the design and theoretical limitations of deep ReLU networks. This foundation paves the way for continued innovation in the field of neural networks, both in theoretical explorations and practical applications.