Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Segment Every Out-of-Distribution Object (2311.16516v4)

Published 27 Nov 2023 in cs.CV

Abstract: Semantic segmentation models, while effective for in-distribution categories, face challenges in real-world deployment due to encountering out-of-distribution (OoD) objects. Detecting these OoD objects is crucial for safety-critical applications. Existing methods rely on anomaly scores, but choosing a suitable threshold for generating masks presents difficulties and can lead to fragmentation and inaccuracy. This paper introduces a method to convert anomaly \textbf{S}core \textbf{T}o segmentation \textbf{M}ask, called S2M, a simple and effective framework for OoD detection in semantic segmentation. Unlike assigning anomaly scores to pixels, S2M directly segments the entire OoD object. By transforming anomaly scores into prompts for a promptable segmentation model, S2M eliminates the need for threshold selection. Extensive experiments demonstrate that S2M outperforms the state-of-the-art by approximately 20% in IoU and 40% in mean F1 score, on average, across various benchmarks including Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Autoencoders for unsupervised anomaly segmentation in brain mr images: a comparative study. Medical Image Analysis, 69:101952, 2021.
  2. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
  3. Fishyscapes: A benchmark for safe semantic segmentation in autonomous driving. In proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  4. Anomaly detection in autonomous driving: A survey. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2022.
  5. Segmentmeifyoucan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104.14812, 2021a.
  6. Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In Proceedings of the ieee/cvf international conference on computer vision, pages 5128–5137, 2021b.
  7. Lara: A light and anti-overfitting retraining approach for unsupervised anomaly detection. arXiv preprint arXiv:2310.05668, 2023.
  8. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  9. Open set semantic segmentation with statistical test and adaptive threshold. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2020.
  10. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  11. Pixel-wise anomaly detection in complex driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16918–16927, 2021.
  12. Few-shot adaptive detection of objects of concern using generative models with negative retraining. In 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pages 528–535. IEEE, 2021.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In Proceedings of the 37th International Conference on Machine Learning, pages 3145–3153. PMLR, 2020.
  15. Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
  16. Atta: Anomaly-aware test-time adaptation for out-of-distribution detection in segmentation. arXiv preprint arXiv:2309.05994, 2023.
  17. Densehybrid: Hybrid anomaly detection for dense open-set recognition. In European Conference on Computer Vision, pages 500–517. Springer, 2022.
  18. A brief survey on semantic segmentation with deep learning. Neurocomputing, 406, 2020.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016a.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016b.
  21. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  22. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018.
  23. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606, 2018.
  24. N-pad : Neighboring pixel-based industrial anomaly detection, 2022.
  25. Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scene segmentation, 2021.
  26. Segment anything. arXiv:2304.02643, 2023.
  27. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018.
  28. Enhancing the reliability of out-of-distribution image detection in neural networks, 2020.
  29. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  30. Detecting the unexpected via image resynthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2152–2161, 2019.
  31. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464–21475, 2020.
  32. Residual pattern learning for pixel-wise out-of-distribution detection in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1151–1161, 2023.
  33. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  34. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086–7096, 2022.
  35. Anomaly detection through latent space restoration using vector quantized variational autoencoders. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1764–1767, 2021.
  36. Evaluating bayesian deep learning methods for semantic segmentation, 2019.
  37. Rba: Segmenting unknown regions rejected by all. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 711–722, 2023.
  38. Out-of-distribution detection for automotive perception, 2021.
  39. Lost and found: detecting small road hazards for self-driving vehicles. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1099–1106. IEEE, 2016.
  40. Unmasking anomalies in road-scene segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4037–4046, 2023.
  41. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
  42. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666, 2019.
  43. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015a.
  44. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015b.
  45. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262–7272, 2021.
  46. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. In European Conference on Computer Vision, pages 246–263. Springer, 2022.
  47. Redesigning out-of-distribution detection on 3d medical images. In International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, pages 126–135. Springer, 2023.
  48. Detectron2. https://github.com/facebookresearch/detectron2, 2019.
  49. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021.
  50. A survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443–58469, 2020.
  51. Fast segment anything, 2023.
  52. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881–6890, 2021.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets