Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data (2403.06538v1)

Published 11 Mar 2024 in cs.RO, cs.CV, and eess.IV

Abstract: Reflective surfaces present a persistent challenge for reliable 3D mapping and perception in robotics and autonomous systems. However, existing reflection datasets and benchmarks remain limited to sparse 2D data. This paper introduces the first large-scale 3D reflection detection dataset containing more than 50,000 aligned samples of multi-return Lidar, RGB images, and 2D/3D semantic labels across diverse indoor environments with various reflections. Textured 3D ground truth meshes enable automatic point cloud labeling to provide precise ground truth annotations. Detailed benchmarks evaluate three Lidar point cloud segmentation methods, as well as current state-of-the-art image segmentation networks for glass and mirror detection. The proposed dataset advances reflection detection by providing a comprehensive testbed with precise global alignment, multi-modal data, and diverse reflective objects and materials. It will drive future research towards reliable reflection detection. The dataset is publicly available at http://3dref.github.io

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
  2. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017.
  3. The reflectance field map: Mapping glass and specular surfaces in dynamic environments. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8393–8399. IEEE, 2023.
  4. Vector: A versatile event-centric benchmark for multi-sensor slam. IEEE Robotics and Automation Letters, 7(3):8217–8224, 2022.
  5. Reflective noise filtering of large-scale point cloud using multi-position lidar sensing data. Remote Sensing, 13(16):3058, 2021.
  6. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  7. Hesai Group. QT64 — 64-Channel Short-Range Lidar — HESAI Technology — hesaitech.com. https://www.hesaitech.com/product/qt64/, 2023. [Accessed 27-07-2023].
  8. Learning semantic associations for mirror detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5941–5950, 2022.
  9. Zero-shot learning for reflection removal of single 360-degree image. In European Conference on Computer Vision, pages 533–548. Springer, 2022.
  10. Enhanced boundary learning for glass-like object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15859–15868, 2021.
  11. Efficient mirror detection via multi-level heterogeneous learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 790–798, 2023.
  12. Detection and mapping of specular surfaces using multibounce lidar returns. Optics Express, 31(4):6370–6388, 2023.
  13. Tgsnet: Multi-field feature fusion for glass region segmentation using transformers. Mathematics, 11(4):843, 2023.
  14. Symmetry-aware transformer-based mirror detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 935–943, 2023.
  15. insta360. Insta360 ONE RS 1-Inch 360 Edition – Dual 1-Inch Sensor 6K 360 Camera — insta360.com. https://www.insta360.com/product/insta360-oners/1inch-360, 2023. [Accessed 27-07-2023].
  16. Asynchronous multiple lidar-inertial odometry using point-wise inter-lidar uncertainty propagation. IEEE Robotics and Automation Letters, 2023.
  17. Detection and purging of specular reflective and transparent object influences in 3d range measurements. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42:377–384, 2017.
  18. Progressive mirror detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3697–3705, 2020.
  19. Rich context aggregation with reflection prior for glass surface detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13415–13424, 2021.
  20. Exploiting semantic relations for glass surface detection. Advances in Neural Information Processing Systems, 35:22490–22504, 2022a.
  21. Depth-aware glass surface detection with cross-modal context mining. arXiv preprint arXiv:2206.11250, 2022b.
  22. Don’t hit me! glass detection in real-world scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3687–3696, 2020.
  23. Glass segmentation using intensity and spectral polarization cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12622–12631, 2022.
  24. A survey on recent reflective detection methods in simultaneous localization and mapping for robot applications. In 2023 6th International Symposium on Autonomous Systems (ISAS), pages 1–6. IEEE, 2023.
  25. Inc. Ouster. ouster os0-128 rev6 datasheet. https://data.ouster.io/downloads/datasheets/datasheet-rev06-v2p5-os0.pdf, 2023. [Accessed 27-07-2023].
  26. The newer college dataset: Handheld lidar, inertial and vision with ground truth. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4353–4360. IEEE, 2020.
  27. Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5135–5142. IEEE, 2020.
  28. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576, 2015.
  29. Trosd: A new rgb-d dataset for transparent and reflective object segmentation in practice. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  30. Mirror3d: Depth refinement for mirror surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15990–15999, 2021.
  31. Searching efficient 3d architectures with sparse point-voxel convolution. In European conference on computer vision, pages 685–702. Springer, 2020.
  32. Glass recognition and map optimization method for mobile robot based on boundary guidance. Chinese Journal of Mechanical Engineering, 36(1):1–12, 2023.
  33. Livox Technology. Specs - Avia LiDAR sensor - Livox — livoxtech.com. https://www.livoxtech.com/avia/specs, 2023. [Accessed 27-07-2023].
  34. Lidar-based glass detection for improved occupancy grid mapping. Sensors, 21(7):2263, 2021.
  35. Depth-aware cnn for rgb-d segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 135–150, 2018.
  36. Multi-sensor fusion glass detection for robot navigation and mapping. In 2018 WRC Symposium on Advanced Robotics and Automation (WRC SARA), pages 184–188. IEEE, 2018.
  37. Where is my mirror? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8809–8818, 2019.
  38. Pcseg: An open source point cloud segmentation codebase. https://github.com/PJLab-ADG/PCSeg, 2023.
  39. Reflection removal for large-scale 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4597–4605, 2018.
  40. Virtual point removal for large-scale 3d point clouds with multiple glass planes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):729–744, 2019.
  41. Mapping with reflection-detection and utilization of reflection in 3d lidar scans. In 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pages 27–33. IEEE, 2020.
  42. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.
  43. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9939–9948, 2021.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. 3DRef: 3D Dataset and Benchmark for Reflection Detection