Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference (2211.11435v3)

Published 21 Nov 2022 in cs.LG and cs.CV

Abstract: Whereas the ability of deep networks to produce useful predictions has been amply demonstrated, estimating the reliability of these predictions remains challenging. Sampling approaches such as MC-Dropout and Deep Ensembles have emerged as the most popular ones for this purpose. Unfortunately, they require many forward passes at inference time, which slows them down. Sampling-free approaches can be faster but suffer from other drawbacks, such as lower reliability of uncertainty estimates, difficulty of use, and limited applicability to different types of tasks and data. In this work, we introduce a sampling-free approach that is generic and easy to deploy, while producing reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost. It is predicated on training the network to produce the same output with and without additional information about it. At inference time, when no prior information is given, we use the network's own prediction as the additional information. We then take the distance between the predictions with and without prior information as our uncertainty measure. We demonstrate our approach on several classification and regression tasks. We show that it delivers results on par with those of Ensembles but at a much lower computational cost.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Uncertainty Estimation Using a Single Deep Deterministic Neural Network. In International Conference on Machine Learning, 9690–9700.
  2. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning. In International Conference on Learning Representations.
  3. Deep Ordinal Regression with Label Diversity. In International Conference on Pattern Recognition, 2740–2747.
  4. Weight Uncertainty in Neural Network. In International Conference on Machine Learning, 1613–1622.
  5. Brier, G. W.; et al. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly weather review, 78(1): 1–3.
  6. Graphnorm: A principled approach to accelerating graph neural network training. In International Conference on Machine Learning.
  7. Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation. Pattern Recognition, 140: 325–331.
  8. Shapenet: An Information-Rich 3D Model Repository. In arXiv Preprint.
  9. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In arXiv Preprint.
  10. Aleatory or epistemic? Does it matter? Structural safety, 31(2): 105–112.
  11. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In arXiv Preprint.
  12. Drela, M. 1989. XFOIL: An Analysis and Design System for Low Reynolds Number Airfoils. In Conference on Low Reynolds Number Aerodynamics, 1–12.
  13. Masksembles for Uncertainty Estimation. In Conference on Computer Vision and Pattern Recognition.
  14. DEBOSH: Deep Bayesian Shape Optimization. In arXiv Preprint.
  15. Quantile regression: estimation and simulation, Volume 2, volume 216. John Wiley & Sons.
  16. Gal, Y. 2016. Uncertainty in Deep Learning. Ph.D. thesis, University of Cambridge.
  17. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In International Conference on Machine Learning, 1050–1059.
  18. Lightweight Probabilistic Deep Networks. In Conference on Computer Vision and Pattern Recognition.
  19. Graves, A. 2011. Practical variational inference for neural networks. Advances in Neural Information Processing Systems, 24.
  20. On Calibration of Modern Neural Networks. In International Conference on Learning Representations.
  21. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, 770–778.
  22. Natural adversarial examples. In Conference on Computer Vision and Pattern Recognition, 15262–15271.
  23. Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. In International Conference on Machine Learning, 1861–1869.
  24. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning.
  25. Airfoil Section Characteristics as Affected by Variations of the Reynolds Number. Report-National Advisory Committee for Aeronautics, 227: 577–611.
  26. OpenFOAM: A C++ Library for Complex Physics Simulations. In International workshop on coupled methods in numerical dynamics.
  27. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In Advances in Neural Information Processing Systems.
  28. Adam: A Method for Stochastic Optimisation. In International Conference on Learning Representations.
  29. Variational Dropout and the Local Parameterization Trick. Advances in Neural Information Processing Systems, 28.
  30. The CIFAR-10 Dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 55(5).
  31. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems.
  32. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 2278–2324.
  33. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. In Advances in Neural Information Processing Systems.
  34. A General Framework for Uncertainty Estimation in Deep Learning. IEEE Robotics and Automation Letters, 5(2): 3153–3160.
  35. Mackay, D. J. 1995. Bayesian Neural Networks and Density Networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 354(1): 73–80.
  36. Regression Prior Networks. In arXiv Preprint.
  37. Predictive Uncertainty Estimation via Prior Networks. In Advances in Neural Information Processing Systems.
  38. Training-free uncertainty estimation for dense regression: Sensitivity as a surrogate. In AAAI Conference on Artificial Intelligence, 10042–10050.
  39. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations.
  40. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In Conference on Computer Vision and Pattern Recognition, 5425–5434.
  41. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv Preprint.
  42. Deep Deterministic Uncertainty for Semantic Segmentation. In arXiv Preprint.
  43. Reading Digits in Natural Images with Unsupervised Feature Learning. In Advances in Neural Information Processing Systems.
  44. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Conference on Computer Vision and Pattern Recognition, 427–436.
  45. Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation. In Conference on Computer Vision and Pattern Recognition, 2931–2940.
  46. On the Practicality of Deterministic Epistemic Uncertainty. In arXiv Preprint.
  47. Random Features for Large-Scale Kernel Machines. In Advances in Neural Information Processing Systems, 1177–1184.
  48. Meshsdf: Differentiable Iso-Surface Extraction. In Advances in Neural Information Processing Systems.
  49. Imagenet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3): 211–252.
  50. Feed-Forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers. In International Conference on Learning Representations.
  51. Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification. In arXiv Preprint.
  52. Learning Local Feature Descriptors Using Convex Optimisation. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  53. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations.
  54. Frequentist Uncertainty Estimates for Deep Learning. In arXiv Preprint.
  55. Single-Model Uncertainties for Deep Learning. In Advances in Neural Information Processing Systems.
  56. ROC curve, lift chart and calibration plot. Advances in methodology and Statistics, 3(1): 89–108.
  57. Natural-Parameter Networks: A Class of Probabilistic Neural Networks. Advances in Neural Information Processing Systems, 29.
  58. (Probabilistic Pixel-Adaptive Refinement Networks). In Conference on Computer Vision and Pattern Recognition, 11642–11651.
  59. Approximating the Bethe Partition Function. In Uncertainty in Artificial Intelligence.
  60. Batchensemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. In International Conference on Learning Representations.
  61. Fashion-Mnist: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. In arXiv Preprint.
  62. Deep Layer Aggregation. In Conference on Computer Vision and Pattern Recognition, 2403–2412.
  63. Understanding Deep Learning (still) Requires Rethinking Generalization. Communications of the ACM, 64(3): 107–115.
  64. Reducing uncertainty in undersampled MRI reconstruction with active acquisition. In Conference on Computer Vision and Pattern Recognition, 2049–2058.
  65. Age Progression/regression by Conditional Adversarial Autoencoder. In Conference on Computer Vision and Pattern Recognition.
  66. Cartoon Face Recognition: A Benchmark Dataset. In Proceedings of the 28th ACM international conference on multimedia, 2264–2272.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets