Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions (2303.14115v2)

Published 24 Mar 2023 in cs.CV

Abstract: Deep neural networks for scene perception in automated vehicles achieve excellent results for the domains they were trained on. However, in real-world conditions, the domain of operation and its underlying data distribution are subject to change. Adverse weather conditions, in particular, can significantly decrease model performance when such data are not available during training.Additionally, when a model is incrementally adapted to a new domain, it suffers from catastrophic forgetting, causing a significant drop in performance on previously observed domains. Despite recent progress in reducing catastrophic forgetting, its causes and effects remain obscure. Therefore, we study how the representations of semantic segmentation models are affected during domain-incremental learning in adverse weather conditions. Our experiments and representational analyses indicate that catastrophic forgetting is primarily caused by changes to low-level features in domain-incremental learning and that learning more general features on the source domain using pre-training and image augmentations leads to efficient feature reuse in subsequent tasks, which drastically reduces catastrophic forgetting. These findings highlight the importance of methods that facilitate generalized features for effective continual learning algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Dissecting the high-frequency bias in convolutional neural networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 863–871, 2021.
  2. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), 9 2018.
  3. Are transformers more robust than cnns? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 26831–26843. Curran Associates, Inc., 2021.
  4. Albumentations: Fast and flexible image augmentations. Information, 11(2), 2020.
  5. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020.
  6. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  7. Incremental learning in semantic segmentation from image labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4371–4381, June 2022.
  8. Modeling the background for incremental learning in semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 9230–9239, 2 2020.
  9. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  10. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
  11. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  12. Similarity and matching of neural network representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5656–5668. Curran Associates, Inc., 2021.
  13. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16712–16721, June 2022.
  14. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  15. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  16. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html, 2012.
  17. Self-Supervised Training Enhances Online Continual Learning. In British Machine Vision Conference (BMVC), 2021.
  18. Multi-domain incremental learning for semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 761–771, January 2022.
  19. Faster autoaugment: Learning augmentation strategies using backpropagation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, pages 1–16. Springer, 2020.
  20. Remind your neural network to prevent catastrophic forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  21. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
  22. Pavel Iakubovskii. Segmentation models pytorch. https://github.com/qubvel/segmentation_models.pytorch, 2019.
  23. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448–456, Lille, France, 07–09 Jul 2015. PMLR.
  24. Effects of architectures on continual semantic segmentation. arXiv preprint arXiv:2302.10718, 2023.
  25. Causes of catastrophic forgetting in class-incremental semantic segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 56–73, December 2022.
  26. Continual learning for class- and domain-incremental semantic segmentation. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 1345–1351, 2021.
  27. Continual unsupervised domain adaptation for semantic segmentation, 2020.
  28. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  29. Class-incremental learning for semantic segmentation re-using neither old data nor old labels. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–8. IEEE, 2020.
  30. Similarity of neural network representations revisited. In 36th International Conference on Machine Learning, ICML 2019, volume 2019-June, pages 6156–6175, 5 2019.
  31. Timothée Lesort. Continual feature selection: Spurious features in continual learning. arXiv preprint arXiv:2203.01012, 2022.
  32. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018.
  33. Rehearsal-free continual learning over small non-i.i.d. batches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  34. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  35. Continual unsupervised domain adaptation for semantic segmentation using a class-specific transfer, 2022.
  36. An empirical investigation of the role of pre-training in lifelong learning, 2021.
  37. Incremental learning techniques for semantic segmentation. Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019, pages 3205–3212, 7 2019.
  38. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1114–1124, 6 2021.
  39. An efficient domain-incremental learning approach to drive in all weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3001–3011, 2022.
  40. Architecture matters in continual learning, 2022.
  41. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2021.
  42. What is being transferred in transfer learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 512–523. Curran Associates, Inc., 2020.
  43. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv:1908.01091, 2019.
  44. Continual normalization: Rethinking batch normalization for online continual learning. In International Conference on Learning Representations, 2022.
  45. PMLR. Barlow twins: Self-supervised learning via redundancy reduction, 2021.
  46. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021.
  47. iCaRL: incremental classifier and representation learning. In CVPR, 2017.
  48. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  49. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 19(1):263–272, 2018.
  50. ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
  51. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  52. Continual unsupervised domain adaptation for semantic segmentation by online frequency domain style transfer. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2881–2888, 2021.
  53. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  54. RTFormer: Efficient design for real-time semantic segmentation with transformer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  55. Deep high-resolution representation learning for visual recognition. TPAMI, 2019.
  56. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, Oct. 2020. Association for Computational Linguistics.
  57. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173–15184, 2020.
  58. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  59. Ace: Adapting to changing environments for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  60. Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
  61. Segformer: Simple and efficient design for semantic segmentation with transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12077–12090. Curran Associates, Inc., 2021.
  62. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
  63. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. International Journal of Computer Vision, 129:3051–3068, 2021.
  64. Self-training for class-incremental semantic segmentation. IEEE Transactions on Neural Networks and Learning Systems, pages 1–12, 2022.
  65. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3987–3995, International Convention Centre, Sydney, Australia, 8 2017. PMLR.
  66. Are all layers created equal?, 2019.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now