Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Abstract: This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the $L<sup>2(P)$ norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

• The paper shows that C∞-smooth activations overcome ReLU's uniform convergence limitations in high dimensions by yielding hierarchy-adaptive rates.
• Smooth DNNs leveraging activations like SiLU, GELU, and Mish with residual designs exhibit provably faster convergence and robust statistical performance.
• Empirical and theoretical results confirm that smooth activations significantly reduce estimation errors in tasks such as high-dimensional regression and counterfactual inference.

Mitigating the Curse of Dimensionality: Uniform Convergence of Deep Neural Networks via Smooth Activations

Introduction

This paper provides a rigorous and comprehensive analysis of the uniform convergence behavior of deep neural networks (DNNs) using smooth activations—specifically SiLU, GELU, and Mish—with both feedforward and residual architectures. Unlike much of the prior focus on $L^2(P)$ minimax-optimality, the work centers uniform guarantees, which underpin downstream reliability in statistically rigorous applications such as individualized recommendations, confidence bands, and high-stakes scientific modeling. The authors establish and characterize a fundamental statistical limitation of least-squares estimators using ReLU activations: even with optimal $L^2(P)$ rates, uniform convergence rates suffer strongly from the curse of dimensionality. The paper demonstrates that introducing $C^\infty$-smooth activations and exploiting architectural advances such as ResNets fundamentally changes the statistical landscape, yielding provably faster uniform convergence rates that adapt to the underlying hierarchical structure of the target function.

The Curse of Dimensionality for ReLU: Uniform Convergence Breakdown

A central theoretical result is the establishment of a lower bound for the uniform convergence rate of least-squares ReLU estimators. Under benign settings—including bounded, H\"older-smooth target functions and i.i.d. sampling—the expected $L^\infty$ estimation error cannot converge faster than $n^{-1/(d+1)}$ in $d$ dimensions, regardless of the target's smoothness. This curse-of-dimensionality lower bound holds despite the possibility of simultaneous minimax-optimal $L^2(P)$ rates, revealing a substantial risk for critical applications where worst-case guarantees are essential.

Figure 1: ReLU and its non-$C^\infty$ variants exhibit limited smoothness, in contrast to the $C^\infty$ variants (SiLU, GELU, Mish) that enable high-order approximation.

The authors demonstrate, from an interpolation-inequality perspective, that the lack of sufficient smoothness (piecewise affine, non-$C^1$) in ReLU-activated networks precludes fast rates for the supremum norm. Even with superior $L^2(P)$0 behavior, the non-smoothness provides no mechanism to exploit higher-order structure, making classical tools like Gagliardo–Nirenberg inequalities inapplicable except with ineffective low-order exponents.

Smooth DNNs and Architectural Implications

The authors propose a paradigm shift to DNNs with $L^2(P)$1-smooth activations, grouping SiLU (Swish), GELU, and Mish as representative examples. These activations are all Pfaffian, possess high-order derivative control, and can be implemented efficiently. In terms of architecture, both feedforward and deep residual networks are included in the framework.

Figure 2: Architectures of an FNN (left) and residual blocks with $L^2(P)$2 (right); residual connections stabilize gradient flow and increase effective capacity.

The mathematical analysis leverages the regularity of these activations to yield structural and complexity advantages. Residual connections further enable the construction of deep networks with well-conditioned optimization landscapes.

Statistical Properties: Pseudo-Dimension, Approximation, and Smoothness

Through detailed bounds on the pseudo-dimension, the authors establish that the statistical complexity of these models remains controlled (polylogarithmic in width and depth, polynomial in dimension) even with high-order smooth activations. Crucially, the networks can approximate Sobolev functions and hierarchical composition models (which capture the compositional structure observed in practical data) with non-asymptotic error bounds matching the best available rates in the literature for deep architectures.

Moreover, the smooth DNNs can be constructed so that their high-order H\"older and Sobolev norms are tightly controlled, with explicit factorial-type bounds in terms of the model parameters. This regularity is what enables tight interpolation inequalities to transfer improved performance from $L^2(P)$3 to $L^2(P)$4 loss.

Uniform Convergence Results

For a wide range of statistical estimation tasks—including Huber, least-squares, quantile, and logistic regression—the authors derive nonasymptotic, rate-optimal uniform convergence bounds for smooth DNN estimators. These rates adapt sharply to the hierarchical compositional complexity $L^2(P)$5 of the target, rather than the ambient data dimension $L^2(P)$6, and do not display the curse-of-dimensionality degradation observed with ReLU networks.

Critically, smooth DNNs also exhibit robustness to heavy-tailed noise (via Huber loss), a property previously only established for $L^2(P)$7 error, extending guarantees to the uniform metric.

Figure 3: Side-by-side boxplots of the $L^2(P)$8 estimation errors illustrate all architectures, with smooth DNNs consistently lower than ReLU.

Figure 4: Side-by-side boxplots of the $L^2(P)$9 estimation errors; the uniform convergence gap between ReLU and smooth DNNs is immediately apparent.

Empirically, simulation experiments (including high-dimensional regression with heavy-tailed noise) confirm that SiLU FNNs and ResNets achieve substantially lower estimation errors (both in $C^\infty$0 and $C^\infty$1), and also exhibit higher convergence exponents, compared to ReLU FNNs. Residual architectures further improve both error and consistency.

Application: Counterfactual Ozone Modeling

The theory's practical relevance is demonstrated via the application of smooth-activated ResNet quantile estimators to continental-scale ozone concentration analysis. Uniform convergence is essential for credible counterfactual inference under climate-driven covariate shifts.

Figure 5: Overall counterfactual difference by SiLU ResNets in ozone concentration—spatially smooth, interpretable, and robust maps.

The results show that SiLU ResNets yield counterfactual surfaces with markedly lower spatial roughness and a reduced fraction of spurious extremes, relative to ReLU FNNs, supporting their use in high-stakes geoscientific prediction settings.

Implications, Limitations, and Future Directions

This work justifies, both theoretically and empirically, the transition toward $C^\infty$2-activated, residual DNNs for statistical tasks where uniform reliability is critical. The framework bridges the statistical optimality gap between classical function estimation and contemporary deep learning, aligning advances in neural architecture with uniform error control.

The results suggest several directions for future work:

• Extension to adaptive architectures that exploit unknown, latent hierarchical structure.
• Investigation of uniform convergence in broader regimes (e.g., under distributional shift, causal inference, sequential data).
• Joint optimization of architecture and regularity for automated uniform reliability.
• Explicit exploration of uniform generalization in foundation models and large-scale pretraining.

Conclusion

This paper establishes, for the first time, a sharp contrast between ReLU and smooth activation DNNs in terms of uniform convergence, formally identifying the former's curse-of-dimensionality bottleneck in the supremum norm setting. By importing $C^\infty$3-smooth activations, supported by deep residual designs, the authors construct estimators with provably optimal, hierarchy-adaptive uniform convergence rates, strong empirical performance, and robustness to heavy-tailed noise. These results ground the empirical practice of using smooth, deep, residual networks for statistically rigorous inference in high-dimensional and high-stakes applications.