Estimating label quality and errors in semantic segmentation data via any model (2307.05080v1)
Abstract: The labor-intensive annotation process of semantic segmentation datasets is often prone to errors, since humans struggle to label every pixel correctly. We study algorithms to automatically detect such annotation errors, in particular methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in order to ensure a high-quality training/evaluation dataset, which is critical in sensitive applications such as medical imaging and autonomous vehicles. Widely applicable, our label quality scores rely on probabilistic predictions from a trained segmentation model -- any model architecture and training procedure can be utilized. Here we study 7 different label quality scoring methods used in conjunction with a DeepLabV3+ or a FPN segmentation model to detect annotation errors in a version of the SYNTHIA dataset. Precision-recall evaluations reveal a score -- the soft-minimum of the model-estimated likelihoods of each pixel's annotated class -- that is particularly effective to identify images that are mislabeled, across multiple types of annotation error.
- Combating noisy labels in object detection datasets. arXiv preprint arXiv:2211.13993, 2022.
- Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation, 2021.
- Encoder-decoder with atrous separable convolution for semantic image segmentation, 2018.
- Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, 2019.
- The cityscapes dataset for semantic urban scene understanding. CoRR, abs/1604.01685, 2016. URL http://arxiv.org/abs/1604.01685.
- Activelab: Active learning with re-labeling by multiple annotators. In ICLR Workshop on Trustworthy ML, 2023.
- Imperfect segmentation labels: How much do they matter?, 2018.
- Iakubovskii, P. Segmentation models pytorch. https://github.com/qubvel/segmentation_models.pytorch, 2019.
- Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, 2018.
- Segment anything, 2023.
- Annotation error detection: Analyzing the past and present for a more coherent future. arXiv preprint arXiv:2206.02280, 2022.
- Model-agnostic label quality scoring to detect real-world label errors. In ICML DataPerf Workshop, 2022.
- Feature pyramid networks for object detection, 2017.
- Identifying mislabeled instances in classification datasets. In International Joint Conference on Neural Networks. IEEE, jul 2019.
- Can we trust bounding box annotations for object detection? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4813–4822, 2022.
- Learning with noisy labels. In Advances in Neural Information Processing Systems, 2013.
- Augmentation strategies for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
- Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021a.
- Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021b.
- OpenCV. Opencv (open source computer vision library), 2023. URL https://opencv.org/. Open Source Computer Vision Library.
- The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3234–3243, 2016. doi: 10.1109/CVPR.2016.352.
- Automated detection of label errors in semantic segmentation datasets via deep learning and uncertainty quantification, 2022.
- The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e0118432, 2015.
- Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
- Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080, 2014.
- Identifying incorrect annotations in multi-label classification data. ICLR Workshop on Trustworthy ML, 2023.
- An analysis of the impact of annotation errors on the accuracy of deep learning for cell segmentation. In International Conference on Medical Imaging with Deep Learning, pp. 1251–1267. PMLR, 2022.
- Detecting label errors in token classification data. In NeurIPS Workshop on Interactive Learning for Natural Language Processing, 2022.
- Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
- Disentangling human error from the ground truth in segmentation of medical images. In Advances in Neural Information Processing Systems, 2020.
- Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, 2018.