Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 97 tok/s
Gemini 2.5 Pro 58 tok/s Pro
GPT-5 Medium 38 tok/s
GPT-5 High 37 tok/s Pro
GPT-4o 101 tok/s
GPT OSS 120B 466 tok/s Pro
Kimi K2 243 tok/s Pro
2000 character limit reached

An Analytic Solution to Covariance Propagation in Neural Networks (2403.16163v1)

Published 24 Mar 2024 in cs.LG, cs.AI, and stat.ML

Abstract: Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Uncertainty propagation through deep neural networks. In Proc. Interspeech 2015, pages 3561–3565.
  2. Propagation of uncertainty through multilayer perceptrons for robust automatic speech recognition. In Proc. Interspeech 2011, pages 461–464.
  3. Analytic expressions for probabilistic moments of pl-dnn with Gaussian input. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9099–9107.
  4. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 1613–1622. JMLR.org.
  5. Tractable inference for complex stochastic processes. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI’98, page 33–42, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  6. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57, Los Alamitos, CA, USA. IEEE Computer Society.
  7. Daunizeau, J. (2017). Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables. arXiv preprint arXiv:1703.00091.
  8. Laplace Redux - Effortless Bayesian Deep Learning. In Advances in Neural Information Processing Systems, volume 34, pages 20089–20103. Curran Associates, Inc.
  9. On the sensitivity of adversarial robustness to input data distributions. In International Conference on Learning Representations.
  10. Variational learning in nonlinear Gaussian belief networks. Neural Computation, 11(1):193–213.
  11. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1050–1059. JMLR.org.
  12. Lightweight probabilistic deep networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3369–3378.
  13. Table of Integrals, Series, and Products. Academic Press, 5th edition edition.
  14. Graves, A. (2011). Practical variational inference for neural networks. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 24, page 2348–2356. Curran Associates, Inc.
  15. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321–1330. PMLR.
  16. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026–1034.
  17. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415.
  18. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 1861–1869. JMLR.org.
  19. Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge University Press, Cambridge.
  20. The UCI Machine Learning Repository.
  21. Variational dropout and the local reparameterization trick. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 28, page 2575–2583. Curran Associates, Inc.
  22. MNIST handwritten digit database.
  23. Object recognition with gradient-based learning. In Forsyth, D., Mundy, J., di Gesu, V., and Cipolla, R., editors, Shape, Contour and Grouping in Computer Vision, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pages 319–345. Springer Verlag.
  24. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153–3160.
  25. Decoupled weight decay regularization. In International Conference on Learning Representations.
  26. MacKay, D. J. C. (1992). A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472.
  27. MacKay, D. J. C. (2002). Information Theory, Inference & Learning Algorithms. Cambridge University Press, USA.
  28. A simple baseline for Bayesian uncertainty in deep learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13153–13164, Red Hook, NY, USA. Curran Associates Inc.
  29. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
  30. Uncertainty propagation for dropout-based Bayesian neural networks. Neural Networks, 144:394–406.
  31. Minka, T. P. (2001). A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, USA. AAI0803033.
  32. Neal, R. M. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56(1):71–113.
  33. How many perturbations break this model? Evaluating robustness beyond adversarial accuracy. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26583–26598. PMLR.
  34. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Francisco (CA).
  35. On the distribution of penultimate activations of classification networks. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, pages 1141–1151. PMLR. ISSN: 2640-3498.
  36. Shriver, D. (2022). Increasing the Applicability of Verification Tools for Neural Networks. PhD thesis, USA.
  37. Striving for simplicity: The all convolutional net. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
  38. Intriguing properties of neural networks. In International Conference on Learning Representations.
  39. Deterministic variational inference for robust Bayesian neural networks. In International Conference on Learning Representations.
Citations (3)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets