Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PAGER: A Framework for Failure Analysis of Deep Regression Models (2309.10977v2)

Published 20 Sep 2023 in cs.LG and stat.ML

Abstract: Safe deployment of AI models requires proactive detection of failures to prevent costly errors. To this end, we study the important problem of detecting failures in deep regression models. Existing approaches rely on epistemic uncertainty estimates or inconsistency w.r.t the training data to identify failure. Interestingly, we find that while uncertainties are necessary they are insufficient to accurately characterize failure in practice. Hence, we introduce PAGER (Principled Analysis of Generalization Errors in Regressors), a framework to systematically detect and characterize failures in deep regressors. Built upon the principle of anchored training in deep models, PAGER unifies both epistemic uncertainty and complementary manifold non-conformity scores to accurately organize samples into different risk regimes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1134–1144, 2021.
  2. Predicting the generalization gap in deep models using anchoring. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4393–4397. IEEE, 2022.
  3. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274–19289, 2022.
  4. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022.
  5. Artificial intelligence in dermatology: a primer. Journal of Investigative Dermatology, 140(8):1504–1512, 2020.
  6. Yu Huang and Yue Chen. Autonomous driving with deep learning: A survey of state-of-art technologies. arXiv preprint arXiv:2006.06091, 2020.
  7. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  8. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  9. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  10. Bayesian deep ensembles via the neural tangent kernel. Advances in Neural Information Processing Systems, 33:1010–1022, 2020.
  11. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927–14937, 2020.
  12. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=eGLdVRvvfQ.
  13. Data-SUITE: Data-centric identification of in-distribution incongruous examples. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 19467–19496, 17–23 Jul 2022a. URL https://proceedings.mlr.press/v162/seedat22a.html.
  14. Single model uncertainty estimation via stochastic data centering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=j0J9upqN5va.
  15. Learning to extrapolate: A transductive approach. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=lid14UkLPd4.
  16. Predicting out-of-domain generalization with local manifold smoothness. arXiv preprint arXiv:2207.02093, 2022.
  17. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=WvOGCEAQhxl.
  18. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning, pages 1617–1629. PMLR, 2021.
  19. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJlQfnCqKX.
  20. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15069–15078, 2021.
  21. Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022b. URL https://openreview.net/forum?id=qC2BwvfaNdd.
  22. Estimating epistemic uncertainty of graph neural networks. In Data Centric Machine Learning Workshop @ ICML, 2023.
  23. Predictive inference with feature conformal prediction. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0uRm1YmFTu.
  24. Virtual library of simulation experiments. https://www.sfu.ca/~ssurjano/index.html. Accessed: 2023-05-01.
  25. Delve datasets. https://www.cs.toronto.edu/~delve/data/datasets.html. Accessed: 2023-05-11.
  26. Boston housing. https://scikit-learn.org/1.0/modules/generated/sklearn.datasets.load_boston.html. Accessed: 2023-05-11.
  27. Ailerons datsets. https://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html. Accessed: 2023-05-11.
  28. How reliable is your regression model’s uncertainty under real-world distribution shifts? arXiv preprint arXiv:2302.03679, 2023.
  29. Pattern Recognition and Machine Learning. J. Electronic Imaging, 16(4):049901, 2007.
  30. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
  31. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, pages 9690–9700. PMLR, 2020.
  32. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.10108, 2020.
  33. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
  34. Algorithmic learning in a random world, volume 29. Springer, 2005.
  35. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018.
  36. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361–3376, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now