Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation (2403.12505v2)

Published 19 Mar 2024 in cs.CV

Abstract: This paper addresses an interesting yet challenging problem -- source-free unsupervised domain adaptation (SFUDA) for pinhole-to-panoramic semantic segmentation -- given only a pinhole image-trained model (i.e., source) and unlabeled panoramic images (i.e., target). Tackling this problem is nontrivial due to the semantic mismatches, style discrepancies, and inevitable distortion of panoramic images. To this end, we propose a novel method that utilizes Tangent Projection (TP) as it has less distortion and meanwhile slits the equirectangular projection (ERP) with a fixed FoV to mimic the pinhole images. Both projections are shown effective in extracting knowledge from the source model. However, the distinct projection discrepancies between source and target domains impede the direct knowledge transfer; thus, we propose a panoramic prototype adaptation module (PPAM) to integrate panoramic prototypes from the extracted knowledge for adaptation. We then impose the loss constraints on both predictions and prototypes and propose a cross-dual attention module (CDAM) at the feature level to better align the spatial and channel characteristics across the domains and projections. Both knowledge extraction and transfer processes are synchronously updated to reach the best performance. Extensive experiments on the synthetic and real-world benchmarks, including outdoor and indoor scenarios, demonstrate that our method achieves significantly better performance than prior SFUDA methods for pinhole-to-panoramic adaptation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Deep learning for omnidirectional vision: A survey and new perspectives. arXiv preprint arXiv:2205.10468, 2022.
  2. Self-supervised augmentation consistency for adapting semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15384–15394, 2021.
  3. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017.
  4. Source-free domain adaptation for image segmentation. Medical Image Analysis, 82:102617, 2022.
  5. Progressive feature alignment for unsupervised domain adaptation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 627–636, 2019a.
  6. Frozen is better than learning: A new design of prototype-based classifier for semantic segmentation. Available at SSRN 4617170.
  7. Uncertainty teacher with dense focal loss for semi-supervised medical image segmentation. Computers in Biology and Medicine, 149:106034, 2022.
  8. Clip is also a good teacher: A new learning framework for inductive zero-shot semantic segmentation. arXiv preprint arXiv:2310.02296, 2023.
  9. Domain adaptation for semantic segmentation with maximum squares loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2090–2099, 2019b.
  10. Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6829–6839, 2019.
  11. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  12. Tangent images for mitigating spherical distortion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12426–12434, 2020.
  13. Francois Fleuret et al. Uncertainty reduction for model adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9613–9623, 2021.
  14. Simt: Handling open-set noise for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7032–7041, 2022.
  15. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. ArXiv, abs/1612.02649, 2016.
  16. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
  17. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9924–9935, 2022.
  18. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34:3635–3649, 2021.
  19. Segment anything. arXiv preprint arXiv:2304.02643, 2023a.
  20. Segment anything. arXiv preprint arXiv:2304.02643, 2023b.
  21. Generalize then adapt: Source-free domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7046–7056, 2021.
  22. Bidirectional learning for domain adaptation of semantic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6929–6938, 2019.
  23. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2801–2810, 2022.
  24. Pano-sfmlearner: Self-supervised multi-task learning of depth and semantics in panoramic videos. IEEE Signal Processing Letters, 28:832–836, 2021a.
  25. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1215–1224, 2021b.
  26. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2502–2511, 2019.
  27. Densepass: Dense panoramic semantic segmentation via unsupervised domain adaptation with attention-augmented context exchange. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2766–2772. IEEE, 2021.
  28. Pixmatch: Unsupervised domain adaptation via pixelwise consistency training. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12430–12440, 2021.
  29. Image to image translation for domain adaptation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  30. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3764–3773, 2020.
  31. Learning from synthetic data: Addressing domain shift for semantic segmentation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3752–3761, 2018.
  32. Unsupervised domain adaptation for semantic segmentation via self-supervision. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 2747–2750. IEEE, 2021.
  33. Unsupervised model adaptation for continual semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, pages 2593–2601, 2021.
  34. Learning to adapt structured output space for semantic segmentation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7472–7481, 2018.
  35. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  36. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517–2526, 2019a.
  37. Dada: Depth-aware domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7364–7373, 2019b.
  38. Domain adaptive semantic segmentation with self-supervised depth estimation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8495–8505, 2021a.
  39. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568–578, 2021b.
  40. Adversarial co-training for semantic segmentation over medical images. Computers in biology and medicine, 157:106736, 2023.
  41. Source free domain adaptation for semantic segmentation via distribution transfer and adaptive class-balanced self-training. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2022.
  42. Pass: Panoramic annular semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 21(10):4171–4185, 2019.
  43. Omnisupervised omnidirectional semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 2020.
  44. Source data-free unsupervised domain adaptation for semantic segmentation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2233–2242, 2021.
  45. Sofa: Source-data-free feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 474–483, 2021.
  46. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13834–13844, 2021.
  47. Deeppanocontext: Panoramic 3d scene understanding with holistic scene context graph and relation-based optimization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12612–12621, 2021a.
  48. Transfer beyond the field of view: Dense panoramic semantic segmentation via unsupervised domain adaptation. IEEE Transactions on Intelligent Transportation Systems, 2021b.
  49. Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16917–16927, 2022a.
  50. Behind every domain there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation. arXiv preprint arXiv:2207.11860, 2022b.
  51. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12414–12424, 2021c.
  52. Category anchor-guided unsupervised domain adaptation for semantic segmentation. Advances in neural information processing systems, 32, 2019.
  53. Curriculum domain adaptation for semantic segmentation of urban scenes. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2039–2049, 2017.
  54. Source-free open compound domain adaptation in semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 32(10):7019–7032, 2022.
  55. Uncertainty-aware deep co-training for semi-supervised medical image segmentation. Computers in Biology and Medicine, 149:106051, 2022a.
  56. Transformer-cnn cohort: Semi-supervised semantic segmentation by the best of both students. arXiv preprint arXiv:2209.02178, 2022b.
  57. Distilling efficient vision transformers from cnns for semantic segmentation. arXiv preprint arXiv:2310.07265, 2023a.
  58. Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18687–18698, 2023b.
  59. Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation. arXiv preprint arXiv:2303.14360, 2023c.
  60. A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11720–11730, 2023.
  61. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289–305, 2018.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now