Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 97 tok/s
Gemini 2.5 Pro 58 tok/s Pro
GPT-5 Medium 38 tok/s
GPT-5 High 37 tok/s Pro
GPT-4o 101 tok/s
GPT OSS 120B 466 tok/s Pro
Kimi K2 243 tok/s Pro
2000 character limit reached

Probabilistic computation and uncertainty quantification with emerging covariance (2305.19265v3)

Published 30 May 2023 in cs.LG, cs.NE, math.ST, and stat.TH

Abstract: Building robust, interpretable, and secure AI system requires quantifying and representing uncertainty under a probabilistic perspective to mimic human cognitive abilities. However, probabilistic computation presents significant challenges for most conventional artificial neural network, as they are essentially implemented in a deterministic manner. In this paper, we develop an efficient probabilistic computation framework by truncating the probabilistic representation of neural activation up to its mean and covariance and construct a moment neural network that encapsulates the nonlinear coupling between the mean and covariance of the underlying stochastic network. We reveal that when only the mean but not the covariance is supervised during gradient-based learning, the unsupervised covariance spontaneously emerges from its nonlinear coupling with the mean and faithfully captures the uncertainty associated with model predictions. Our findings highlight the inherent simplicity of probabilistic computation by seamlessly incorporating uncertainty into model prediction, paving the way for integrating it into large-scale AI systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. The neural processes underlying perceptual decision making in humans: recent progress and future directions. Journal of Physiology-Paris, 109(1-3):27–37, 2015.
  2. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021.
  3. A survey of uncertainty in deep neural networks. arXiv e-prints, 2021.
  4. Ben Goertzel. Human-level artificial general intelligence and the possibility of a technological singularity: A reaction to ray kurzweil’s the singularity is near, and mcdermott’s critique of kurzweil. Artificial Intelligence, 171(18):1161–1173, 2007.
  5. Mark Sprevak. Two kinds of information processing in cognition. Review of Philosophy and Psychology, 11(3):591–611, 2020.
  6. The computer and the brain. Yale university press, 2012.
  7. Dominik Seuß. Bridging the gap between explainable ai and uncertainty quantification to enhance trustability. arXiv preprint, 2021.
  8. Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553):452–459, 2015.
  9. An introduction to variational methods for graphical models. Machine learning, 37:183–233, 1999.
  10. Sequential Monte Carlo methods in practice. Springer, 2001.
  11. Variational bayesian inference with stochastic search. arXiv preprint, 2012.
  12. Auto-encoding variational bayes. arXiv preprint, 2013.
  13. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  14. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  15. Bayesian uncertainty estimation for batch normalized deep networks. In International Conference on Machine Learning, pages 4907–4916. PMLR, 2018.
  16. Dropconnect is effective in modeling uncertainty of bayesian deep networks. Scientific reports, 11(1):1–14, 2021.
  17. Stimulus-dependent variability and noise correlations in cortical mt neurons. Proceedings of the National Academy of Sciences, 110(32):13162–13167, 2013.
  18. The structures and functions of correlations in neural population codes. Nature Reviews Neuroscience, 23:551–567, 2022.
  19. On a gaussian neuronal field model. NeuroImage, 52(3):913–933, 2010.
  20. Crispin W Gardiner et al. Handbook of stochastic methods, volume 3. springer Berlin, 1985.
  21. Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143–195, 1999.
  22. Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013.
  23. Learning multiple layers of features from tiny images. 2009.
  24. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  25. Matus Telgarsky. Benefits of depth in neural networks. In Conference on learning theory, pages 1517–1539. PMLR, 2016.
  26. Yaroslav Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html, 2, 2011.
  27. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2016.
  28. Intriguing properties of neural networks. arXiv preprint, 2013.
  29. Explaining and harnessing adversarial examples. stat, 1050:20, 2015.
  30. Uci machine learning repository, 2007.
  31. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning, pages 1861–1869. PMLR, 2015.
  32. Sampling-free variational inference for neural networks with multiplicative activation noise. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28–October 1, 2021, Proceedings, pages 33–47. Springer, 2022.
  33. Deterministic variational inference for robust bayesian neural networks. arXiv preprint, 2018.
  34. Mitigating evasion attacks to deep neural networks via region-based classification. In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 278–287, 2017.
  35. Weak pairwise correlations imply strongly correlated network states in a neural population. Nature, 440(7087):1007–1012, 2006.
  36. Correlated fluctuations in strongly coupled binary networks beyond equilibrium. Physical Review X, 6(3):031024, 2016.
  37. Sampling-free variational inference of bayesian neural networks by variance backpropagation. In Uncertainty in Artificial Intelligence, pages 563–573. PMLR, 2020.
  38. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  39. Neural correlations, population coding and computation. Nature Reviews Neuroscience, 7(5):358–366, 2006.
  40. Deep learning. MIT press, 2016.
  41. Correlation between neural spike trains increases with firing rate. Nature, 448(7155):802–806, 2007.
  42. Dynamics of moment neuronal networks. Physical Review E, 73(4):041906, 2006.
  43. Toward spike-based stochastic neural computing. arXiv preprint, 2023.
  44. Dynamics of bump attractors in neural circuits with emergent spatial correlations. arXiv preprint, 2022.
  45. Yang Qi. An efficient numerical algorithm for the moment neural activation. arXiv preprint, 2022.
  46. Unsupervised risk estimation using only conditional independence structure. Advances in Neural Information Processing Systems, 29, 2016.
  47. Adam: A method for stochastic optimization. arXiv preprint, 2014.
  48. Very deep convolutional networks for large-scale image recognition. arXiv preprint, 2014.
  49. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
  50. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint, 2017.
  51. Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
  52. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688, 2011.
  53. Real analysis, volume 32. Macmillan New York, 1988.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now