Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks (2310.12079v2)

Published 18 Oct 2023 in stat.ML and cs.LG

Abstract: Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential equations. However, these results do not a priori tell us anything about "ordinary" unshaped networks, where the activation is unchanged as the network size grows. In this article, we find similar differential equation based asymptotic characterization for two types of unshaped networks. Firstly, we show that the following two architectures converge to the same infinite-depth-and-width limit at initialization: (i) a fully connected ResNet with a $d{-1/2}$ factor on the residual branch, where $d$ is the network depth. (ii) a multilayer perceptron (MLP) with depth $d \ll$ width $n$ and shaped ReLU activation at rate $d{-1/2}$. Secondly, for an unshaped MLP at initialization, we derive the first order asymptotic correction to the layerwise correlation. In particular, if $\rho_\ell$ is the correlation at layer $\ell$, then $q_t = \ell2 (1 - \rho_\ell)$ with $t = \frac{\ell}{n}$ converges to an SDE with a singularity at $t=0$. These results together provide a connection between shaped and unshaped network architectures, and opens up the possibility of studying the effect of normalization methods and how it connects with shaping activation functions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp.  4782–4887. PMLR, 2022.
  2. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. arXiv preprint arXiv:2205.01445, 2022.
  3. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  4. Deep learning: a statistical viewpoint. Acta numerica, 30:87–201, 2021.
  5. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023.
  6. Kernel methods for deep learning. In Advances in Neural Information Processing Systems (NeurIPS), pp.  342–350, 2009.
  7. Neural signature kernels as infinite-width-depth-limits of controlled resnets. arXiv preprint arXiv:2303.17671, 2023.
  8. Gradient descent finds global minima of deep neural networks. In Int. Conf. Machine Learning (ICML), pp.  1675–1685. PMLR, 2019.
  9. Markov processes: characterization and convergence. John Wiley & Sons, 2009.
  10. Optimal signal propagation in resnets through residual scaling. arXiv preprint arXiv:2305.07715, 2023.
  11. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33:14820–14830, 2020.
  12. Finite depth and width corrections to the neural tangent kernel. In Int. Conf. Learning Representations (ICLR), 2019a.
  13. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, pp.  1–36, 2019b.
  14. Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. Transactions on Machine Learning Research, 2022.
  15. Soufiane Hayou. Commutative width and depth scaling in deep neural networks, 2023.
  16. Width and depth limits commute in residual networks. arXiv preprint arXiv:2302.00453, 2023.
  17. Stable ResNet. In Int. Conf. Artificial Intelligence and Statistics (AISTATS), pp.  1324–1332. PMLR, 2021.
  18. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. IEEE Int. Conf. Computer Vision, pp.  1026–1034, 2015.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  20. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. pmlr, 2015.
  21. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Information Processing Systems (NeurIPS), 2018.
  22. Depth degeneracy in neural networks: Vanishing angles in fully connected relu networks on initialization. arXiv preprint arXiv:2302.09712, 2023.
  23. O. Kallenberg. Foundations of Modern Probability. Probability theory and stochastic modelling. Springer, 2021. ISBN 9783030618728.
  24. Deep neural networks as gaussian processes. In Int. Conf. Learning Representations (ICLR), 2018.
  25. The future is log-gaussian: Resnets and their infinite-depth-and-width limit at initialization. Advances in Neural Information Processing Systems, 34, 2021.
  26. Geometric dyson brownian motion and the free log-normal for minor of products of random matrices. In Preparation, 2024.
  27. The neural covariance sde: Shaped infinite depth-and-width networks at initialization. arXiv preprint arXiv:2206.02768, 2022.
  28. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv preprint arXiv:2110.01765, 2021.
  29. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018.
  30. Towards training without depth limits: Batch normalization without gradient explosion, 2023.
  31. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peerj-cs.103. URL https://doi.org/10.7717/peerj-cs.103.
  32. Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 1995.
  33. Precise characterization of the prior predictive distribution of deep relu networks. Advances in Neural Information Processing Systems, 34, 2021.
  34. The shaped transformer: Attention models in the infinite depth-and-width limit. arXiv preprint arXiv:2306.17759, 2023.
  35. Mean field analysis of neural networks: A law of large numbers, 2018.
  36. Multidimensional diffusion processes, volume 233. Springer Science & Business Media, 1997.
  37. Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes, 2019.
  38. Feature learning in infinite-width neural networks. In Int. Conf. Machine Learning (ICML), 2021.
  39. Tensor programs vi: Feature learning in infinite-depth neural networks, 2023.
  40. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. arXiv preprint arXiv:2203.08120, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets