Papers
Topics
Authors
Recent
Search
2000 character limit reached

UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather

Published 8 Apr 2024 in cs.CV | (2404.05145v1)

Abstract: LiDAR semantic segmentation (LSS) is a critical task in autonomous driving and has achieved promising progress. However, prior LSS methods are conventionally investigated and evaluated on datasets within the same domain in clear weather. The robustness of LSS models in unseen scenes and all weather conditions is crucial for ensuring safety and reliability in real applications. To this end, we propose UniMix, a universal method that enhances the adaptability and generalizability of LSS models. UniMix first leverages physically valid adverse weather simulation to construct a Bridge Domain, which serves to bridge the domain gap between the clear weather scenes and the adverse weather scenes. Then, a Universal Mixing operator is defined regarding spatial, intensity, and semantic distributions to create the intermediate domain with mixed samples from given domains. Integrating the proposed two techniques into a teacher-student framework, UniMix efficiently mitigates the domain gap and enables LSS models to learn weather-robust and domain-invariant representations. We devote UniMix to two main setups: 1) unsupervised domain adaption, adapting the model from the clear weather source domain to the adverse weather target domain; 2) domain generalization, learning a model that generalizes well to unseen scenes in adverse weather. Extensive experiments validate the effectiveness of UniMix across different tasks and datasets, all achieving superior performance over state-of-the-art methods. The code will be released.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Metareg: Towards domain generalization using meta-regularization. Advances in Neural Information Processing Systems, 31, 2018.
  2. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297–9307, 2019.
  3. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  4. Refign: Align and refine for adaptation of semantic segmentation to adverse conditions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3174–3184, 2023.
  5. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11580–11590, 2021.
  6. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
  7. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In Advances in Visual Computing, pages 207–222. Springer, 2020.
  8. Benchmarking robustness of 3d object detection to common corruptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1022–1032, 2023.
  9. Survey on lidar perception in adverse weather conditions. arXiv preprint arXiv:2304.06312, 2023.
  10. Condaformer: Disassembled transformer with local structure enhancement for 3d point cloud understanding. Advances in Neural Information Processing Systems, 36, 2024.
  11. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3354–3361, 2012.
  12. Train one, generalize to all: Generalizable semantic segmentation from single-scene to all adverse scenes. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2275–2284, 2023.
  13. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9224–9232, 2018.
  14. The distribution with size of aggregate snowflakes. Journal of Atmospheric Sciences, 15(5):452–461, 1958.
  15. Pct: Point cloud transformer. Computational Visual Media, 7:187–199, 2021.
  16. Fog simulation on real lidar point clouds for 3d object detection in adverse weather. In CVPR, 2021.
  17. Lidar snowfall simulation for robust 3d object detection. In CVPR, 2022.
  18. Shruti Jadon. A survey of loss functions for semantic segmentation. In IEEE conference on Computational Intelligence in Bioinformatics and Computational Biology, pages 1–7, 2020.
  19. Lidar light scattering augmentation (lisa): Physics-based simulation of adverse weather conditions for 3d object detection. arXiv preprint arXiv:2107.07004, 2021.
  20. Single domain generalization for lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17587–17598, 2023.
  21. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation, pages 9338–9345, 2023a.
  22. Lasermix for semi-supervised lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21705–21715, 2023b.
  23. Domain transfer for semantic segmentation of lidar data using deep neural networks. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 8263–8270, 2020.
  24. Domain generalization with adversarial feature learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5400–5409, 2018a.
  25. Pointcnn: Convolution on x-transformed points. Advances in Neural Information Processing Systems, 31, 2018b.
  26. Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  27. Both style and fog matter: Cumulative domain adaptation for semantic foggy scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18922–18931, 2022.
  28. Rangenet++: Fast and accurate lidar semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4213–4220, 2019.
  29. A quantitative analysis of point clouds from automotive lidars exposed to artificial rain and fog. Atmosphere, 12(6):738, 2021.
  30. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5648–5656, 2016.
  31. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017a.
  32. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30, 2017b.
  33. Influences of weather phenomena on automotive laser radar systems. Advances in radio science, 9:49–60, 2011.
  34. Unsupervised domain adaptation in lidar semantic segmentation with self-supervision and gated adapters. In IEEE International Conference on Robotics and Automation, pages 2649–2655, 2022.
  35. Cosmix: Compositional semantic mix for domain adaptation in 3d lidar segmentation. In European Conference on Computer Vision, pages 586–602. Springer, 2022.
  36. Compositional semantic mix for domain adaptation in point cloud segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
  37. Walking your lidog: A journey through multiple domains for lidar semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 196–206, 2023b.
  38. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  39. Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2530–2539, 2018.
  40. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
  41. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019.
  42. Adversarial discriminative domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  43. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517–2526, 2019.
  44. Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud. In IEEE International Conference on Robotics and Automation, pages 1887–1893, 2018.
  45. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In IEEE International Conference on Robotics and Automation, pages 4376–4382, 2019.
  46. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1912–1920, 2015.
  47. Polarmix: A general data augmentation technique for lidar point clouds. Advances in Neural Information Processing Systems, 35:11035–11048, 2022a.
  48. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2795–2803, 2022b.
  49. 3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9382–9392, 2023.
  50. Pcl: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7097–7107, 2022.
  51. Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15363–15373, 2021.
  52. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12414–12424, 2021.
  53. Category anchor-guided unsupervised domain adaptation for semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
  54. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601–9610, 2020.
  55. Domain generalization via entropy regularization. Advances in Neural Information Processing Systems, 33:16096–16107, 2020.
  56. epointda: An end-to-end simulation-to-real domain adaptation framework for lidar point cloud segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3500–3509, 2021.
  57. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  58. Understanding the robustness of 3d object detection with bird’s-eye-view representations in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21600–21610, 2023.
  59. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982–5991, 2019.
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up