Papers
Topics
Authors
Recent
Search
2000 character limit reached

Enabling Uncertainty Estimation in Iterative Neural Networks

Published 25 Mar 2024 in cs.AI | (2403.16732v3)

Abstract: Turning pass-through network architectures into iterative ones, which use their own output as input, is a well-known approach for boosting performance. In this paper, we argue that such architectures offer an additional benefit: The convergence rate of their successive outputs is highly correlated with the accuracy of the value to which they converge. Thus, we can use the convergence rate as a useful proxy for uncertainty. This results in an approach to uncertainty estimation that provides state-of-the-art estimates at a much lower computational cost than techniques like Ensembles, and without requiring any modifications to the original iterative model. We demonstrate its practical value by embedding it in two application domains: road detection in aerial images and the estimation of aerodynamic properties of 2D and 3D shapes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563–3593, 2014.
  2. Allaire, G. A Review of Adjoint Methods for Sensitivity Analysis, Uncertainty Quantification and Optimization in Numerical Codes. Ingénieurs de l’Automobile, 836:33–36, July 2015.
  3. Uncertainty Estimation Using a Single Deep Deterministic Neural Network. In International Conference on Machine Learning, pp. 9690–9700, 2020.
  4. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning. In International Conference on Learning Representations, 2020.
  5. Auer, P. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422, 2002.
  6. Attention is All You Need. In Advances in Neural Information Processing Systems, 2017.
  7. Segnet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
  8. Geodesic Convolutional Shape Optimization. In International Conference on Machine Learning, 2018.
  9. Roadtracer: Automatic Extraction of Road Networks from Aerial Images. In Conference on Computer Vision and Pattern Recognition, 2018.
  10. Generalized denoising auto-encoders as generative models. Advances in neural information processing systems, 26, 2013.
  11. Weight Uncertainty in Neural Network. In International Conference on Machine Learning, pp. 1613–1622, 2015.
  12. Human Pose Estimation with Iterative Error Feedback. In Conference on Computer Vision and Pattern Recognition, 2016.
  13. Shapenet: An Information-Rich 3D Model Repository. In arXiv Preprint, 2015.
  14. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, 2018.
  15. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In arXiv Preprint, 2015.
  16. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In arXiv Preprint, 2018.
  17. Double Refinement Network for Efficient Monocular Depth Estimation. In International Conference on Intelligent Robots and Systems, pp.  5889–5894, 2019.
  18. Masksembles for Uncertainty Estimation. In Conference on Computer Vision and Pattern Recognition, 2021.
  19. ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference. arXiv Preprint, 2022a.
  20. Partal: Efficient partial active learning in multi-task visual settings. arXiv preprint arXiv:2211.11546, 2022b.
  21. Fawcett, T. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874, 2006.
  22. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  23. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
  24. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068–7081, 2021.
  25. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In International Conference on Machine Learning, pp. 1050–1059, 2016.
  26. Graves, A. Practical variational inference for neural networks. Advances in Neural Information Processing Systems, 24, 2011.
  27. Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows. In Advances in Neural Information Processing Systems, 2020.
  28. Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision. In Conference on Computer Vision and Pattern Recognition, 2020.
  29. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.
  30. Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. In International Conference on Machine Learning, pp. 1861–1869, 2015.
  31. Graph Neural Networks for the Prediction of Aircraft Surface Pressure Distributions. Aerospace Science and Technology, 137:108268, 2023.
  32. Topology-Preserving Deep Image Segmentation. In Advances in Neural Information Processing Systems, pp. 5658–5669, 2019.
  33. A novelty detection approach to classification. In IJCAI, volume 1, pp.  518–523. Citeseer, 1995.
  34. OpenFOAM: A C++ Library for Complex Physics Simulations. In International workshop on coupled methods in numerical dynamics, 2007.
  35. On autoencoder scoring. In International Conference on Machine Learning, pp. 720–728. PMLR, 2013.
  36. Adam: A Method for Stochastic Optimisation. In International Conference on Learning Representations, 2015.
  37. Variational Dropout and the Local Parameterization Trick. Advances in Neural Information Processing Systems, 28, 2015.
  38. Kumar, S. K. On Weight Initialization in Deep Neural Networks. arXiv Preprint, 2017.
  39. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, 2017.
  40. Comparison of Gradient-Based and Gradient-Enhanced Response-Surface-Based Optimizers. American Institute of Aeronautics and Astronautics Journal, 48(5):981–994, 2010.
  41. Mackay, D. J. Bayesian Neural Networks and Density Networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 354(1):73–80, 1995.
  42. Predictive Uncertainty Estimation via Prior Networks. In Advances in Neural Information Processing Systems, 2018.
  43. Learning to Detect Roads in High-Resolution Aerial Images. In European Conference on Computer Vision, pp.  210–223, 2010.
  44. Mockus, J. Bayesian Approach to Global Optimization: Theory and Applications, volume 37. Springer Science & Business Media, 2012.
  45. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In Conference on Computer Vision and Pattern Recognition, pp. 5425–5434, 2017.
  46. Beyond the Pixel-Wise Loss for Topology-Aware Delineation. In Conference on Computer Vision and Pattern Recognition, pp. 3136–3145, 2018.
  47. Deep Deterministic Uncertainty for Semantic Segmentation. In arXiv Preprint, 2021.
  48. Stacked Hourglass Networks for Human Pose Estimation. In European Conference on Computer Vision, 2016.
  49. Promoting Connectivity of Network-Like Structures by Enforcing Region Separation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5401–5413, 2021.
  50. Adjusting the Ground Truth Annotations for Connectivity-Based Learning to Delineate. IEEE Transactions on Medical Imaging, 41(12):3675–3685, 2022.
  51. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Dhift. In Advances in Neural Information Processing Systems, pp. 13991–14002, 2019.
  52. Deepsdf: Learning Continuous Signed Distance Functions for Shape Representation. In Conference on Computer Vision and Pattern Recognition, 2019.
  53. Automatic Differentiation in Pytorch. In Advances in Neural Information Processing Systems, 2017.
  54. When networks disagree: Ensemble methods for hybrid neural networks. In How We Learn; How We Remember: Toward An Understanding Of Brain And Neural Systems: Selected Papers of Leon N Cooper, pp.  342–358. World Scientific, 1995.
  55. Recurrent Neural Networks for Scene Labelling. In International Conference on Machine Learning, 2014.
  56. Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation. In Conference on Computer Vision and Pattern Recognition, pp. 2931–2940, 2019.
  57. On the Practicality of Deterministic Epistemic Uncertainty. In International Conference on Machine Learning, pp. 17870–17909, 2022.
  58. Improving the expected improvement algorithm. Advances in Neural Information Processing Systems, 30, 2017.
  59. MeshSdf: Differentiable Iso-Surface Extraction. In Advances in Neural Information Processing Systems, 2020.
  60. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Conference on Medical Image Computing and Computer Assisted Intervention, pp.  234–241, 2015.
  61. Response Surface Methods for Efficient Aerodynamic Surrogate Models. In Computational Flight Testing, pp.  113–129. Springer, 2013.
  62. Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electronics Letters, 52(13):1122–1124, 2016.
  63. Image Segmentation with Cascaded Hierarchical Models and Logistic Disjunctive Normal Networks. In International Conference on Computer Vision, 2013.
  64. Multi-Stage Multi-Recursive-Input Fully Convolutional Networks for Neuronal Boundary Detection. In International Conference on Computer Vision, 2017.
  65. Multiscale Centerline Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(7):1327–1341, 2016.
  66. Generalization of iterative sampling in autoencoders. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp.  877–882. IEEE, 2020.
  67. Efficient Multipoint Aerodynamic Design Optimization via Cokriging. Journal of Aircraft, 48(5):1685–1695, 2011.
  68. Auto-Context and Its Applications to High-Level Vision Tasks and 3D Brain Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009.
  69. van Etten, A. Spacenet road detection and routing challenge part ii—apls implementation, 2019.
  70. ROC curve, lift chart and calibration plot. Advances in methodology and Statistics, 3(1):89–108, 2006.
  71. Recurrent U-Net for Resource-Constrained Segmentation. In International Conference on Computer Vision, 2019.
  72. Batchensemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. In International Conference on Learning Representations, 2020.
  73. Empirical Evaluation of Automatically Extracted Road Axes. In Empirical Evaluation Techniques in Computer Vision, pp. 172–187, 1998.
  74. Pad-Net: Multi-Task Guided Prediction-And-Distillation Network for Simultaneous Depth Estimation and Scene Parsing. In Conference on Computer Vision and Pattern Recognition, pp. 675–684, 2018.
  75. Deep Reinforcement Learning for Vessel Centerline Tracing in Multi-Modality 3D Volumes. In Conference on Medical Image Computing and Computer Assisted Intervention, pp.  755–763, 2018a.
  76. Joint task-recursive learning for semantic segmentation and depth estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pp.  235–251, 2018b.
  77. Zhou, Y. Rethinking reconstruction autoencoder-based out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7379–7387, 2022.
  78. Unet++: A Nested U-Net Architecture for Medical Image Segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 31 likes about this paper.

Sign Up for Free