Sample complexity of training non-linear neural networks

Determine the number of training samples required to train a non-linear neural network in order to achieve reliable generalization performance, characterizing the sample complexity as a function of the architecture and data distribution (for example, for deep feed-forward networks with ReLU activation).

Background

The paper highlights that despite extensive practical success, core statistical aspects of neural networks remain insufficiently understood. A central issue is quantifying how the amount of training data impacts generalization for non-linear neural networks.

Prior work has provided various generalization upper bounds with dependencies on norms, depth, or width, and mini-max lower bounds in limited settings (e.g., linear activations or sinusoidal networks). The authors address this question for ReLU feed-forward networks by deriving a minimax lower bound that scales as sqrt(log(d)/n), aligning with recent upper bounds and supported by empirical results. This partially resolves the question in the specific ReLU feed-forward setting, while leaving the broader problem for general non-linear architectures as a fundamental direction.

References

For example, a basic yet very important open question is: how many training samples are needed to train a (non-linear) neural network?

— How many samples are needed to train a deep neural network?  (2405.16696 - Golestaneh et al., 2024) in Section 1, Introduction

A further question is sample complexity: similar to prior finite-width mean-field analyses \citep{li2020learning,mahankali2023beyond,glasgow2025propagation}, our bounds separate feature learning from fixed-kernel methods but do not match the optimal rates suggested by information-exponent analyses \citep{arous2021online,ren2024learning}.

— Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry  (2609.10879 - Zhou et al., 9 Sep 2026) in Section Conclusion

For the dataset generated by the exponential function, the saturation effect is not clearly observed for selected values of training size $N$. In this case, we do not rule out the possibility that providing additional training examples could lead to even better performance.

— Solving In-Table Prediction Problems by Deep Neural Networks with Performance Evaluation Using Synthetic Data  (2609.01262 - Zhao et al., 1 Sep 2026) in Section Results of the experiment, item 2, “How large should be the training size?”