Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation (2404.00979v2)

Published 1 Apr 2024 in cs.CV

Abstract: Existing point cloud semantic segmentation networks cannot identify unknown classes and update their knowledge, due to a closed-set and static perspective of the real world, which would induce the intelligent agent to make bad decisions. To address this problem, we propose a Probability-Driven Framework (PDF) for open world semantic segmentation that includes (i) a lightweight U-decoder branch to identify unknown classes by estimating the uncertainties, (ii) a flexible pseudo-labeling scheme to supply geometry features along with probability distribution features of unknown classes by generating pseudo labels, and (iii) an incremental knowledge distillation strategy to incorporate novel classes into the existing knowledge base gradually. Our framework enables the model to behave like human beings, which could recognize unknown objects and incrementally learn them with the corresponding knowledge. Experimental results on the S3DIS and ScanNetv2 datasets demonstrate that the proposed PDF outperforms other methods by a large margin in both important tasks of open world semantic segmentation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In CVPR, pages 5240–5250, 2023.
  2. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
  3. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2019.
  4. Towards open world recognition. In CVPR, 2015.
  5. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
  6. Deep metric learning for open world semantic segmentation. In ICCV, pages 15333–15342, 2021.
  7. Open-world semantic segmentation for lidar point clouds. In ECCV, pages 318–334, 2022.
  8. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019.
  9. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
  10. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In CVPR, 2018.
  11. Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999.
  12. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1050–1059, New York, New York, USA, 2016. PMLR.
  13. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
  14. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
  15. Scaling out-of-distribution detection for real-world settings. In Proceedings of the 39th International Conference on Machine Learning, pages 8759–8773. PMLR, 2022.
  16. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  17. Randla-net: Efficient semantic segmentation of large-scale point clouds. In CVPR, 2020.
  18. Pointwise convolutional neural networks. In CVPR, 2018.
  19. Exemplar-based open-set panoptic segmentation network. In CVPR, pages 1175–1184, 2021.
  20. Opengan: Open-set recognition via open data generation. In ICCV, pages 813–822, 2021.
  21. Stratified transformer for 3d point cloud segmentation. In CVPR, pages 8500–8509, 2022.
  22. Spherical transformer for lidar-based 3d recognition. In CVPR, pages 17545–17555, 2023.
  23. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS. Curran Associates, Inc., 2017.
  24. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, 2018.
  25. Octree guided cnn with spherical kernels for 3d point clouds. In CVPR, 2019.
  26. Deepgcns: Can gcns go as deep as cnns? In ICCV, 2019.
  27. Open-set semantic segmentation for point clouds via adversarial prototype framework. In CVPR, pages 9425–9434, 2023.
  28. Pointcnn: Convolution on x-transformed points. In NeurIPS. Curran Associates, Inc., 2018.
  29. Learning without forgetting. IEEE TPAMI, 40(12):2935–2947, 2017.
  30. Detecting the unexpected via image resynthesis. In ICCV, 2019.
  31. Vv-net: Voxel vae net with group convolutions for point cloud segmentation. In ICCV, 2019.
  32. Fast point transformer. In CVPR, pages 16949–16958, 2022.
  33. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017a.
  34. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, 2017b.
  35. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS, pages 23192–23204. Curran Associates, Inc., 2022.
  36. Fully-convolutional point networks for large-scale point clouds. In ECCV, 2018.
  37. Octnet: Learning deep 3d representations at high resolutions. In CVPR, 2017.
  38. Tangent convolutions for dense prediction in 3d. In CVPR, 2018.
  39. Attention is all you need. In NeurIPS. Curran Associates, Inc., 2017.
  40. Deep parametric continuous convolutional neural networks. In CVPR, 2018.
  41. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics, 38(5):1–12, 2019.
  42. Energy-based open-world uncertainty modeling for confidence calibration. In ICCV, pages 9302–9311, 2021.
  43. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, 2019.
  44. Point transformer v2: Grouped vector attention and partition-based pooling. In NeurIPS, pages 33330–33342. Curran Associates, Inc., 2022.
  45. Synthesize then compare: Detecting failures and anomalies for semantic segmentation. In ECCV, pages 145–161, Cham, 2020. Springer International Publishing.
  46. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In ECCV, pages 1–19. Springer, 2020.
  47. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In CVPR, 2020.
  48. Pointweb: Enhancing local neighborhood features for point cloud processing. In CVPR, 2019.
  49. Point transformer. In ICCV, pages 16259–16268, 2021.
  50. Learning placeholders for open-set recognition. In CVPR, pages 4401–4410, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Jinfeng Xu (37 papers)
  2. Siyuan Yang (31 papers)
  3. Xianzhi Li (38 papers)
  4. Yuan Tang (37 papers)
  5. Yixue Hao (16 papers)
  6. Long Hu (35 papers)
  7. Min Chen (200 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now