Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Discriminant Distance-Aware Representation on Deterministic Uncertainty Quantification Methods (2402.12664v1)

Published 20 Feb 2024 in cs.LG

Abstract: Uncertainty estimation is a crucial aspect of deploying dependable deep learning models in safety-critical systems. In this study, we introduce a novel and efficient method for deterministic uncertainty estimation called Discriminant Distance-Awareness Representation (DDAR). Our approach involves constructing a DNN model that incorporates a set of prototypes in its latent representations, enabling us to analyze valuable feature information from the input data. By leveraging a distinction maximization layer over optimal trainable prototypes, DDAR can learn a discriminant distance-awareness representation. We demonstrate that DDAR overcomes feature collapse by relaxing the Lipschitz constraint that hinders the practicality of deterministic uncertainty methods (DUMs) architectures. Our experiments show that DDAR is a flexible and architecture-agnostic method that can be easily integrated as a pluggable layer with distance-sensitive metrics, outperforming state-of-the-art uncertainty estimation methods on multiple benchmark problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Uncertainty in the variational information bottleneck. arXiv preprint arXiv:1807.00906, 2018.
  2. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730, 2018.
  3. Training normalizing flows with the information bottleneck for competitive generative classification. Advances in Neural Information Processing Systems, 33:7828–7840, 2020.
  4. Invertible residual networks. In International Conference on Machine Learning, pages 573–582. PMLR, 2019.
  5. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
  6. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356–1367, 2020.
  7. Natural posterior network: Deep bayesian uncertainty for exponential family distributions. arXiv preprint arXiv:2105.04471, 2021.
  8. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  9. Latent discriminant deterministic uncertainty. arXiv preprint arXiv:2207.10130, 2022.
  10. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  11. Contrastive prototype learning with augmented embeddings for few-shot learning. In Uncertainty in Artificial Intelligence, pages 140–150. PMLR, 2021.
  12. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589, 2023.
  13. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  14. Sampling-free variational inference of bayesian neural networks by variance backpropagation. In Uncertainty in Artificial Intelligence, pages 563–573. PMLR, 2020.
  15. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  16. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027, 2019.
  17. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8334–8343, 2021.
  18. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498–7512, 2020.
  19. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153–3160, 2020.
  20. Entropic out-of-distribution detection. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021.
  21. Distinction maximization loss: Efficiently improving classification accuracy, uncertainty estimation, and out-of-distribution detection simply replacing the loss and calibrating. arXiv preprint arXiv:2205.05874, 2022.
  22. Distance-based confidence score for neural network classifiers. arXiv preprint arXiv:1709.09844, 2017.
  23. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
  24. Deep deterministic uncertainty: A simple baseline. arXiv e-prints, pages arXiv–2102, 2021a.
  25. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.11582, 2021b.
  26. Hybrid models with deep and invertible features. In International Conference on Machine Learning, pages 4723–4732. PMLR, 2019.
  27. Practical deep learning with bayesian principles. Advances in neural information processing systems, 32, 2019.
  28. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
  29. Quantifying aleatoric and epistemic uncertainty using density estimation in latent space.
  30. Sampling-free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2931–2940, 2019.
  31. The hidden uncertainty in a neural networks activations. arXiv preprint arXiv:2012.03082, 2020.
  32. On the practicality of deterministic epistemic uncertainty. arXiv preprint arXiv:2107.00649, 2021.
  33. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
  34. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411, 2022.
  35. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pages 9690–9700. PMLR, 2020.
  36. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409, 2021.
  37. Towards open intent discovery for conversational text. arXiv preprint arXiv:1904.08524, 2019.
  38. A discriminative feature learning approach for deep face recognition. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pages 499–515. Springer, 2016.
  39. How good is the bayes posterior in deep neural networks really? In International Conference on Machine Learning, pages 10248–10259. PMLR, 2020.
  40. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566, 2020.
  41. A simple framework for uncertainty in contrastive learning. arXiv preprint arXiv:2010.02038, 2020.
  42. User utterance acquisition for training task-oriented bots: a review of challenges, techniques and opportunities. IEEE Internet Computing, 24(3):30–38, 2020.
  43. Jiaxin Zhang. Modern monte carlo methods for efficient uncertainty quantification and propagation: A survey. Wiley Interdisciplinary Reviews: Computational Statistics, 13(5):e1539, 2021.
  44. On the quantification of image reconstruction uncertainty without training data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2072–2081, 2024.
  45. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1198–1209, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Jiaxin Zhang (105 papers)
  2. Kamalika Das (19 papers)
  3. Sricharan Kumar (11 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets