Papers
Topics
Authors
Recent
Search
2000 character limit reached

AIO2: Online Correction of Object Labels for Deep Learning with Incomplete Annotation in Remote Sensing Image Segmentation

Published 3 Mar 2024 in cs.CV | (2403.01641v1)

Abstract: While the volume of remote sensing data is increasing daily, deep learning in Earth Observation faces lack of accurate annotations for supervised optimization. Crowdsourcing projects such as OpenStreetMap distribute the annotation load to their community. However, such annotation inevitably generates noise due to insufficient control of the label quality, lack of annotators, frequent changes of the Earth's surface as a result of natural disasters and urban development, among many other factors. We present Adaptively trIggered Online Object-wise correction (AIO2) to address annotation noise induced by incomplete label sets. AIO2 features an Adaptive Correction Trigger (ACT) module that avoids label correction when the model training under- or overfits, and an Online Object-wise Correction (O2C) methodology that employs spatial information for automated label modification. AIO2 utilizes a mean teacher model to enhance training robustness with noisy labels to both stabilize the training accuracy curve for fitting in ACT and provide pseudo labels for correction in O2C. Moreover, O2C is implemented online without the need to store updated labels every training epoch. We validate our approach on two building footprint segmentation datasets with different spatial resolutions. Experimental results with varying degrees of building label noise demonstrate the robustness of AIO2. Source code will be available at https://github.com/zhu-xlab/AIO2.git.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources,” IEEE geoscience and remote sensing magazine, vol. 5, no. 4, pp. 8–36, 2017.
  2. J. E. Vargas-Munoz, S. Srivastava, D. Tuia, and A. X. Falcão, “OpenStreetMap: Challenges and opportunities in machine learning and remote sensing,” IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 1, pp. 184–199, 2021.
  3. C. M. Albrecht, F. Marianno, and L. J. Klein, “AutoGeoLabel: Automated label generation for geospatial machine learning,” in 2021 IEEE International Conference on Big Data (Big Data).   IEEE, 2021, pp. 1779–1786.
  4. C. F. Brown, S. P. Brumby, B. Guzder-Williams, T. Birch, S. B. Hyde, J. Mazzariello, W. Czerwinski, V. J. Pasquarella, R. Haertel, S. Ilyushchenko et al., “Dynamic world, near real-time global 10 m land use land cover mapping,” Scientific Data, vol. 9, no. 1, p. 251, 2022.
  5. D. Zanaga, R. Van De Kerchove, W. De Keersmaecker, N. Souverijns, C. Brockmann, R. Quast, J. Wevers, A. Grosu, A. Paccini, S. Vergnaud, O. Cartus, M. Santoro, S. Fritz, I. Georgieva, M. Lesiv, S. Carter, M. Herold, L. Li, N.-E. Tsendbazar, F. Ramoino, and O. Arino, “Esa worldcover 10 m 2020 v100,” Oct. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5571936
  6. K. Karra, C. Kontgis, Z. Statman-Weil, J. C. Mazzariello, M. Mathis, and S. P. Brumby, “Global land use / land cover with sentinel 2 and deep learning,” in 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 2021, pp. 4704–4707.
  7. X. X. Zhu, J. Hu, C. Qiu, Y. Shi, J. Kang, L. Mou, H. Bagheri, M. Haberle, Y. Hua, R. Huang et al., “So2sat lcz42: A benchmark data set for the classification of global local climate zones [software and data sets],” IEEE Geoscience and Remote Sensing Magazine, vol. 8, no. 3, pp. 76–89, 2020.
  8. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021.
  9. C. Liu, C. M. Albrecht, Y. Wang, and X. X. Zhu, “Peaks fusion assisted early-stopping strategy for overhead imagery segmentation with noisy labels,” in 2022 IEEE International Conference on Big Data (Big Data), 2022, pp. 4842–4847.
  10. R. Hänsch and O. Hellwich, “The truth about ground truth: Label noise in human-generated reference data,” in IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.   IEEE, 2019, pp. 5594–5597.
  11. G. Wang, X. Liu, C. Li, Z. Xu, J. Ruan, H. Zhu, T. Meng, K. Li, N. Huang, and S. Zhang, “A noise-robust framework for automatic segmentation of covid-19 pneumonia lesions from ct images,” IEEE Transactions on Medical Imaging, vol. 39, no. 8, pp. 2653–2663, 2020.
  12. P. Li, X. He, M. Qiao, X. Cheng, J. Li, X. Guo, T. Zhou, D. Song, M. Chen, D. Miao, Y. Jiang, and Z. Tian, “Exploring label probability sequence to robustly learn deep convolutional neural networks for road extraction with noisy datasets,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–18, 2022.
  13. L. He, J. Li, C. Liu, and S. Li, “Recent advances on spectral–spatial hyperspectral image classification: An overview and new guidelines,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 3, pp. 1579–1597, 2018.
  14. A. Maiti, S. J. Oude Elberink, and G. Vosselman, “Effect of label noise in semantic segmentation of high resolution aerial images and height data,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. V-2-2022, pp. 275–282, 2022.
  15. W. Shen, Z. Peng, X. Wang, H. Wang, J. Cen, D. Jiang, L. Xie, X. Yang, and Q. Tian, “A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  16. B. Zhang, J. Xiao, Y. Wei, M. Sun, and K. Huang, “Reliability does matter: An end-to-end weakly supervised semantic segmentation approach,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 12 765–12 772.
  17. F. Zhang, Y. Shi, Z. Xiong, W. Huang, and X. X. Zhu, “Pseudo features-guided self-training for domain adaptive semantic segmentation of satellite images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–14, 2023.
  18. Y. Cao and X. Huang, “A coarse-to-fine weakly supervised learning method for green plastic cover segmentation using high-resolution remote sensing images,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 188, pp. 157–176, 2022.
  19. R. Dong, W. Fang, H. Fu, L. Gan, J. Wang, and P. Gong, “High-resolution land cover mapping through learning with noise correction,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  20. Y. Cao and X. Huang, “A full-level fused cross-task transfer learning method for building change detection using noise-robust pretrained networks on crowdsourced labels,” Remote Sensing of Environment, vol. 284, p. 113371, 2023.
  21. S. Liu, K. Liu, W. Zhu, Y. Shen, and C. Fernandez-Granda, “Adaptive early-learning correction for segmentation from noisy annotations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2606–2616.
  22. J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher et al., “A survey of uncertainty in deep neural networks,” Artificial Intelligence Review, pp. 1–77, 2023.
  23. M. A. Brovelli and G. Zamboni, “A new method for the assessment of spatial accuracy and completeness of openstreetmap building footprints,” ISPRS International Journal of Geo-Information, vol. 7, no. 8, p. 289, 2018.
  24. Y. Zhang, Q. Zhou, M. A. Brovelli, and W. Li, “Assessing osm building completeness using population data,” International Journal of Geographical Information Science, vol. 36, no. 7, pp. 1443–1466, 2022.
  25. B. Herfort, S. Lautenbach, J. Porto de Albuquerque, J. Anderson, and A. Zipf, “A spatio-temporal analysis investigating completeness and inequalities of global urban building data in openstreetmap,” Nature Communications, vol. 14, no. 1, p. 3985, 2023.
  26. A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in Advances in neural information processing systems, vol. 30, 2017.
  27. J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu, “Learning with noisy labels revisited: A study using real-world human annotations,” in International Conference on Learning Representations, 2022.
  28. S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus, “Training convolutional networks with noisy labels,” arXiv preprint arXiv:1406.2080, 2014.
  29. A. J. Bekker and J. Goldberger, “Training deep neural-networks based on unreliable labels,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2016, pp. 2682–2686.
  30. J. Goldberger and E. Ben-Reuven, “Training deep neural-networks using a noise adaptation layer,” in International conference on learning representations, 2017.
  31. D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa, “Joint optimization framework for learning with noisy labels,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5552–5560.
  32. G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger, “Identifying mislabeled data using the area under the margin ranking,” Advances in Neural Information Processing Systems, vol. 33, pp. 17 044–17 056, 2020.
  33. E. Malach and S. Shalev-Shwartz, “Decoupling” when to update” from” how to update”,” Advances in neural information processing systems, vol. 30, 2017.
  34. L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in ICML, 2018.
  35. B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
  36. X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?” in International Conference on Machine Learning.   PMLR, 2019, pp. 7164–7173.
  37. J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” in International Conference on Learning Representations, 2020.
  38. P.-F. Zhang, Z. Huang, G. Bai, and X.-S. Xu, “Ideal: High-order-ensemble adaptation network for learning with noisy labels,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 325–333.
  39. S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda, “Early-learning regularization prevents memorization of noisy labels,” Advances in neural information processing systems, vol. 33, pp. 20 331–20 342, 2020.
  40. K. Yi and J. Wu, “Probabilistic end-to-end noise correction for learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7017–7025.
  41. S. E. Reed and H. Lee, “Training deep neural networks on noisy labels with bootstrapping,” in International Conference on Learning Representations 2015 (ICLR 2015), 2015.
  42. Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
  43. Y. Lyu and I. W. Tsang, “Curriculum loss: Robust learning and generalization against label corruption,” arXiv preprint arXiv:1905.10045, 2019.
  44. Y. Liu and H. Guo, “Peer loss functions: Learning from noisy labels without knowing noise rates,” in International conference on machine learning.   PMLR, 2020, pp. 6226–6236.
  45. H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  46. N. Manwani and P. Sastry, “Noise tolerance under risk minimization,” IEEE transactions on cybernetics, vol. 43, no. 3, pp. 1146–1151, 2013.
  47. A. Ghosh, H. Kumar, and P. S. Sastry, “Robust loss functions under label noise for deep neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017.
  48. X. Tai, G. Wang, C. Grecos, and P. Ren, “Coastal image classification under noisy labels,” Journal of Coastal Research, vol. 102, no. SI, pp. 151–156, 2020.
  49. Z. Huang, C. O. Dumitru, Z. Pan, B. Lei, and M. Datcu, “Classification of large-scale high-resolution sar images with deep transfer learning,” IEEE Geoscience and Remote Sensing Letters, vol. 18, no. 1, pp. 107–111, 2020.
  50. B. B. Damodaran, R. Flamary, V. Seguy, and N. Courty, “An entropic optimal transport loss for learning deep neural networks under label noise in remote sensing images,” Computer Vision and Image Understanding, vol. 191, p. 102863, 2020.
  51. Y. Li, Y. Zhang, and Z. Zhu, “Error-tolerant deep learning for remote sensing image scene classification,” IEEE transactions on cybernetics, vol. 51, no. 4, pp. 1756–1768, 2020.
  52. B. Tu, W. Kuang, W. He, G. Zhang, and Y. Peng, “Robust learning of mislabeled training samples for remote sensing image scene classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 5623–5639, 2020.
  53. J. Kang, R. Fernandez-Beltran, X. Kang, J. Ni, and A. Plaza, “Noise-tolerant deep neighborhood embedding for remotely sensed images with label noise,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 2551–2562, 2021.
  54. J. Kang, R. Fernandez-Beltran, P. Duan, X. Kang, and A. J. Plaza, “Robust normalized softmax loss for deep metric learning-based characterization of remote sensing images with label noise,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 10, pp. 8798–8811, 2020.
  55. T. Burgert, M. Ravanbakhsh, and B. Demir, “On the effects of different types of label noise in multi-label remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  56. A. K. Aksoy, M. Ravanbakhsh, and B. Demir, “Multi-label noise robust collaborative learning for remote sensing image classification,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2022.
  57. G. Sumbul and B. Demir, “Generative reasoning integrated label noise robust deep image representation learning,” IEEE Transactions on Image Processing, pp. 1–1, 2023.
  58. E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Convolutional neural networks for large-scale remote-sensing image classification,” IEEE Transactions on geoscience and remote sensing, vol. 55, no. 2, pp. 645–657, 2016.
  59. P. Kaiser, J. D. Wegner, A. Lucchi, M. Jaggi, T. Hofmann, and K. Schindler, “Learning aerial image segmentation from online maps,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 11, pp. 6054–6068, 2017.
  60. N. Ahmed, R. M. Rahman, M. S. G. Adnan, and B. Ahmed, “Dense prediction of label noise for learning building extraction from aerial drone imagery,” International Journal of Remote Sensing, vol. 42, no. 23, pp. 8906–8929, 2021.
  61. P. Li, X. He, M. Qiao, X. Cheng, Z. Li, H. Luo, D. Song, D. Li, S. Hu, R. Li et al., “Robust deep neural networks for road extraction from remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 7, pp. 6182–6197, 2020.
  62. Z. Zhang, W. Guo, M. Li, and W. Yu, “Gis-supervised building extraction with label noise-adaptive fully convolutional neural network,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 12, pp. 2135–2139, 2020.
  63. C. Henry, F. Fraundorfer, and E. Vig, “Aerial road segmentation in the presence of topological label noise,” in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 2336–2343.
  64. K. Malkin, C. Robinson, L. Hou, R. Soobitsky, J. Czawlytko, D. Samaras, J. Saltz, L. Joppa, and N. Jojic, “Label super-resolution networks,” in International Conference on Learning Representations, 2019.
  65. C. Lin, S. Guo, J. Chen, L. Sun, X. Zheng, Y. Yang, and Y. Xiong, “Deep learning network intensification for preventing noisy-labeled samples for remote sensing classification,” Remote Sensing, vol. 13, no. 9, p. 1689, 2021.
  66. C. M. Albrecht, R. Zhang, X. Cui, M. Freitag, H. F. Hamann, L. J. Klein, U. Finkler, F. Marianno, J. Schmude, N. Bobroff et al., “Change detection from remote sensing to guide openstreetmap labeling,” ISPRS International Journal of Geo-Information, vol. 9, no. 7, p. 427, 2020.
  67. J. Sun, J. Liu, L. Hu, Z. Wei, and L. Xiao, “A mutual teaching framework with momentum correction for unsupervised hyperspectral image change detection,” Remote Sensing, vol. 14, no. 4, 2022.
  68. S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” in International Conference on Learning Representations, 2017.
  69. D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio et al., “A closer look at memorization in deep networks,” in International conference on machine learning.   PMLR, 2017, pp. 233–242.
  70. M.-P. Dubuisson and A. K. Jain, “A modified hausdorff distance for object matching,” in Proceedings of 12th international conference on pattern recognition, vol. 1.   IEEE, 1994, pp. 566–568.
  71. S. Jadon, “A survey of loss functions for semantic segmentation,” in 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), 2020, pp. 1–7.
  72. V. Mnih, “Machine learning for aerial image labeling,” Ph.D. dissertation, University of Toronto, 2013.
  73. N. Skuppin, E. J. Hoffmann, Y. Shi, and X. X. Zhu, “Building type classification with incomplete labels,” in IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, 2022, pp. 5844–5847.
  74. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds.   Cham: Springer International Publishing, 2015, pp. 234–241.
  75. M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97.   PMLR, 09–15 Jun 2019, pp. 6105–6114.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces AIO2, a method that adaptively triggers online object-wise label correction to handle incomplete annotations in remote sensing data.
  • It employs an ACT module for optimal correction timing and an O2C module for spatially aware label refinement during model training.
  • Experiments on RS datasets demonstrate AIO2’s robustness, achieving about a 10-point IoU improvement with 30% missing labels.

Exploring AIO2: A Novel Approach for Improving Object-wise Label Correction in Remote Sensing Image Segmentation

In the paper titled "AIO2: Online Correction of Object Labels for Deep Learning with Incomplete Annotation in Remote Sensing Image Segmentation", the authors introduce a sophisticated method designed to address the challenge of working with labels of varying accuracy in remote sensing (RS) imagery data. This new methodology, named Adaptively trIggered Online Object-wise correction (AIO2), is primarily focused on enhancing the process of training models on datasets that exhibit incomplete annotation—a predicament commonly faced in Earth Observation (EO) applications.

The Motivation Behind AIO2

Remote Sensing Image Segmentation is crucial for accurate Earth observation and has a wide range of applications including urban planning, environmental monitoring, and disaster response. The capability of deep learning models to handle complex RS tasks is well-documented. However, their success hinges on the availability of high-quality, accurately labeled datasets, which are often expensive and time-consuming to produce. The AIO2 method proposes a solution to make efficient use of the available data, including those with incomplete labels, by introducing an adaptive, online object-wise correction strategy.

A Dive into AIO2's Core Components

The AIO2 framework comprises two main elements: the Adaptive Correction Trigger (ACT) module and the Online Object-wise label Correction (O2C) module. The ACT module is responsible for determining the optimal timing for label correction during model training, thus avoiding the pitfalls of initiating correction too early or late in the training process. The O2C module, on the other hand, implements an object-level approach to correct labels in a spatially aware manner, offering an improvement over the traditional pixel-wise correction strategies.

These components are designed to work in harmony, with the ACT module leveraging the memorization effects observed during model training to trigger the O2C module at an opportune moment. This synergy ensures that label correction is both timely and relevant, substantially improving the model's learning from noisy, incomplete annotations.

Empirical Validation and Performance

The efficacy of AIO2 was validated through comprehensive experiments conducted on two RS datasets with varying spatial resolutions: one from the Massachusetts area and another covering multiple cities in Germany. These datasets were significantly challenged by incomplete labeling. AIO2 not only demonstrated a remarkable ability to improve segmentation accuracy but also showcased its robustness across different levels of label noise and image resolutions. For instance, on the Massachusetts dataset with a 30% label omission, AIO2 achieved performance gains of about 10 percentage points in Intersection over Union (IoU) compared to training with the original noisy labels.

Implications and Future Directions

The introduction of AIO2 holds significant implications for the field of remote sensing and EO, particularly in scenarios where complete and accurate annotations are hard to come by. By efficiently utilizing datasets with incomplete labels, AIO2 paves the way for more scalable and cost-effective solutions in RS image segmentation tasks.

Going forward, the authors plan to extend the application of AIO2 to tackle other types of label noise, explore its effectiveness in multi-class segmentation tasks, and investigate the potential of employing AIO2 in a multi-round correction approach. These future endeavors aim to further validate the versatility and adaptability of AIO2 across a broader spectrum of RS applications and challenges.

Concluding Thoughts

The AIO2 methodology presents a promising avenue for enhancing the performance of deep learning models in RS image segmentation tasks, especially when faced with the challenge of incomplete annotations. Its innovative use of adaptive triggering for label correction and the incorporation of spatial information at an object-wise level signify major advancements in tackling the prevalent issue of noisy labels in EO datasets. As such, AIO2 exemplifies a significant step forward in the development of data-efficient, robust models for Earth observation, setting a new benchmark for future research in this domain.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 16 likes about this paper.

Sign Up for Free