Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hierarchical Selective Classification (2405.11533v1)

Published 19 May 2024 in cs.LG and cs.CV

Abstract: Deploying deep neural networks for risk-sensitive tasks necessitates an uncertainty estimation mechanism. This paper introduces hierarchical selective classification, extending selective classification to a hierarchical setting. Our approach leverages the inherent structure of class relationships, enabling models to reduce the specificity of their predictions when faced with uncertainty. In this paper, we first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves. Next, we develop algorithms for hierarchical selective classification (which we refer to as "inference rules"), and propose an efficient algorithm that guarantees a target accuracy constraint with high probability. Lastly, we conduct extensive empirical studies on over a thousand ImageNet classifiers, revealing that training regimes such as CLIP, pretraining on ImageNet21k and knowledge distillation boost hierarchical selective performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Knapsack pruning with inner distillation. CoRR, abs/2002.08258, 2020. URL https://arxiv.org/abs/2002.08258.
  2. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. CoRR, abs/2107.07511, 2021. URL https://arxiv.org/abs/2107.07511.
  3. Conformal risk control. CoRR, abs/2208.02814, 2022.
  4. Making better mistakes: Leveraging class hierarchies with deep networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 12503–12512. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.01252. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Bertinetto_Making_Better_Mistakes_Leveraging_Class_Hierarchies_With_Deep_Networks_CVPR_2020_paper.html.
  5. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In NeurIPS, 2022.
  6. Luís Felipe P. Cattelan and Danilo Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks, 2024.
  7. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 2818–2829. IEEE, 2023. doi: 10.1109/CVPR52729.2023.00276. URL https://doi.org/10.1109/CVPR52729.2023.00276.
  8. C. K. Chow. An optimum character recognition system using decision functions. IRE Trans. Electron. Comput., 6(4):247–254, 1957. doi: 10.1109/TEC.1957.5222035. URL https://doi.org/10.1109/TEC.1957.5222035.
  9. A method for improving classification reliability of multilayer perceptrons. IEEE Trans. Neural Networks, 6(5):1140–1147, 1995. doi: 10.1109/72.410358. URL https://doi.org/10.1109/72.410358.
  10. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society, 2009. doi: 10.1109/CVPR.2009.5206848. URL https://doi.org/10.1109/CVPR.2009.5206848.
  11. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 3450–3457. IEEE Computer Society, 2012. doi: 10.1109/CVPR.2012.6248086. URL https://doi.org/10.1109/CVPR.2012.6248086.
  12. On the foundations of noise-free selective classification. J. Mach. Learn. Res., 11:1605–1641, 2010. doi: 10.5555/1756006.1859904. URL https://dl.acm.org/doi/10.5555/1756006.1859904.
  13. EVA: exploring the limits of masked visual representation learning at scale. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 19358–19369. IEEE, 2023. doi: 10.1109/CVPR52729.2023.01855. URL https://doi.org/10.1109/CVPR52729.2023.01855.
  14. Towards better selective classification. In ICLR. OpenReview.net, 2023.
  15. Calibrated selective classification. Trans. Mach. Learn. Res., 2022, 2022.
  16. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=p66AzKi6Xim.
  17. Selective classification for deep neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4878–4887, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html.
  18. Selectivenet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2151–2159. PMLR, 2019. URL http://proceedings.mlr.press/v97/geifman19a.html.
  19. Bias-reduced uncertainty estimation for deep neural classifiers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=SJfb5jCqKm.
  20. Eugen Grycko. Classification with set-valued decision functions. In Otto Opitz, Berthold Lausen, and Rüdiger Klar, editors, Information and Classification, pages 218–224, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-642-50974-2.
  21. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 2017. URL http://proceedings.mlr.press/v70/guo17a.html.
  22. Benchmarking representation learning for natural world image collections. In CVPR, pages 12884–12893. Computer Vision Foundation / IEEE, 2021.
  23. Level selector network for optimizing accuracy-specificity trade-offs. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 1466–1473. IEEE, 2019. doi: 10.1109/ICCVW.2019.00184. URL https://doi.org/10.1109/ICCVW.2019.00184.
  24. No cost likelihood manipulation at test time for making better mistakes in deep networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=193sEnKY1ij.
  25. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992–10002. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00986. URL https://doi.org/10.1109/ICCV48922.2021.00986.
  26. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science, pages 185–201. Springer, 2018. doi: 10.1007/978-3-030-01216-8\_12. URL https://doi.org/10.1007/978-3-030-01216-8_12.
  27. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. CoRR, abs/2310.14770, 2023.
  28. Torchvision the machine-vision package of torch. In ACM Multimedia, pages 1485–1488. ACM, 2010.
  29. George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995. doi: 10.1145/219717.219748. URL https://doi.org/10.1145/219717.219748.
  30. Obtaining well calibrated probabilities using bayesian binning. In AAAI, pages 2901–2907. AAAI Press, 2015.
  31. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021. URL http://proceedings.mlr.press/v139/radford21a.html.
  32. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6517–6525. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.690. URL https://doi.org/10.1109/CVPR.2017.690.
  33. Imagenet-21k pretraining for the masses. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/98f13708210194c475687be6106a3b84-Abstract-round1.html.
  34. LAION-5B: an open large-scale dataset for training next generation image-text models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/a1859debfb3b59d094f3504d5ebb6c25-Abstract-Datasets_and_Benchmarks.html.
  35. To reject or not to reject: that is the question-an answer in case of neural classifiers. IEEE Trans. Syst. Man Cybern. Part C, 30(1):84–94, 2000. doi: 10.1109/5326.827457. URL https://doi.org/10.1109/5326.827457.
  36. Efficientnetv2: Smaller models and faster training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 2021. URL http://proceedings.mlr.press/v139/tan21a.html.
  37. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021. URL http://proceedings.mlr.press/v139/touvron21a.html.
  38. Deit III: revenge of the vit. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, volume 13684 of Lecture Notes in Computer Science, pages 516–533. Springer, 2022. doi: 10.1007/978-3-031-20053-3\_30. URL https://doi.org/10.1007/978-3-031-20053-3_30.
  39. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rkZvSe-RZ.
  40. Jack Valmadre. Hierarchical classification at multiple operating points. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/727855c31df8821fd18d41c23daebf10-Abstract-Conference.html.
  41. Vladimir Vovk. Conditional validity of inductive conformal predictors. CoRR, abs/1209.2673, 2012.
  42. Machine-learning applications of algorithmic randomness. In Ivan Bratko and Saso Dzeroski, editors, Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999, pages 444–453. Morgan Kaufmann, 1999.
  43. Ross Wightman. Pytorch image models. https://github.com/huggingface/pytorch-image-models, 2019.
  44. Resnet strikes back: An improved training procedure in timm. CoRR, abs/2110.00476, 2021. URL https://arxiv.org/abs/2110.00476.
  45. Hierarchical loss for classification. CoRR, abs/1709.01062, 2017. URL http://arxiv.org/abs/1709.01062.
  46. Solving long-tailed recognition with deep realistic taxonomic classifier. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII, volume 12353 of Lecture Notes in Computer Science, pages 171–189. Springer, 2020. doi: 10.1007/978-3-030-58598-3\_11. URL https://doi.org/10.1007/978-3-030-58598-3_11.
  47. Adversarial examples improve image recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 816–825. Computer Vision Foundation / IEEE, 2020a. doi: 10.1109/CVPR42600.2020.00090. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Xie_Adversarial_Examples_Improve_Image_Recognition_CVPR_2020_paper.html.
  48. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10684–10695. Computer Vision Foundation / IEEE, 2020b. doi: 10.1109/CVPR42600.2020.01070. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Xie_Self-Training_With_Noisy_Student_Improves_ImageNet_Classification_CVPR_2020_paper.html.
  49. Billion-scale semi-supervised learning for image classification. CoRR, abs/1905.00546, 2019. URL http://arxiv.org/abs/1905.00546.
  50. Metaformer baselines for vision. IEEE Trans. Pattern Anal. Mach. Intell., 46(2):896–912, 2024. doi: 10.1109/TPAMI.2023.3329173. URL https://doi.org/10.1109/TPAMI.2023.3329173.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets