Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

R3DS: Reality-linked 3D Scenes for Panoramic Scene Understanding (2403.12301v1)

Published 18 Mar 2024 in cs.CV

Abstract: We introduce the Reality-linked 3D Scenes (R3DS) dataset of synthetic 3D scenes mirroring the real-world scene arrangements from Matterport3D panoramas. Compared to prior work, R3DS has more complete and densely populated scenes with objects linked to real-world observations in panoramas. R3DS also provides an object support hierarchy, and matching object sets (e.g., same chairs around a dining table) for each scene. Overall, R3DS contains 19K objects represented by 3,784 distinct CAD models from over 100 object categories. We demonstrate the effectiveness of R3DS on the Panoramic Scene Understanding task. We find that: 1) training on R3DS enables better generalization; 2) support relation prediction trained with R3DS improves performance compared to heuristically calculated support; and 3) R3DS offers a challenging benchmark for future work on panoramic scene understanding.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Scan2CAD: Learning CAD model alignment in RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  2. Rearrangement: A challenge for embodied AI. arXiv preprint arXiv:2011.01975, 2020.
  3. Matterport3D: Learning from RGB-D data in indoor environments. In Proceedings of the International Conference on 3D Vision (3DV), pages 667–676. IEEE, 2017.
  4. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
  5. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21126–21136, 2022.
  6. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5828–5839, 2017.
  7. PanoContext-Former: Panoramic total scene understanding with a transformer. arXiv preprint arXiv:2305.12497, 2023.
  8. 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics. arXiv preprint arXiv:2011.09127, 2020a.
  9. 3D-Future: 3D Furniture shape with TextURE. arXiv preprint arXiv:2009.09633, 2020b.
  10. SceneNN: A scene meshes dataset with annotations. In Proceedings of the International Conference on 3D Vision (3DV), pages 92–101. IEEE, 2016.
  11. Tero Karras. Maximizing parallelism in the construction of bvhs, octrees, and k-d trees. In Proceedings of the Fourth ACM SIGGRAPH / Eurographics Conference on High-Performance Graphics, pages 33–37. Eurographics Association, 2012.
  12. OpenRooms: An end-to-end open framework for photorealistic indoor scene datasets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  13. Parsing ikea objects: Fine pose estimation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2992–2999, 2013.
  14. CAD-estate: Large-scale CAD model annotation in RGB videos. arXiv preprint arXiv:2306.09011, 2023.
  15. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34:12013–12026, 2021.
  16. Habitat-Matterport 3D dataset (hm3d): 1000 large-scale 3D environments for embodied AI. arXiv preprint arXiv:2109.08238, 2021.
  17. Shrenik Sadalgi. Wayfair’s 3D Model API. https://www.aboutwayfair.com/tech-innovation/wayfairs-3d-model-api, 2016. [Online; accessed 15-Nov-2023].
  18. iGibson, a simulation environment for interactive tasks in large realistic scenes. In Proceedings of the International Conference on Intelligent Robots and Systems (IROS), 2021a.
  19. iGibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In Proceedings of the International Conference on Intelligent Robots and Systems (IROS), pages 7520–7527. IEEE, 2021b.
  20. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019.
  21. Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1047–1056, 2019.
  22. Pix3D: Dataset and methods for single-image 3D shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2974–2983, 2018.
  23. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251–266, 2021.
  24. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision (IJCV), 118(2):172–193, 2016.
  25. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG), 38(4):1–15, 2019.
  26. Objectnet3D: A large scale database for 3D object recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 160–176. Springer, 2016.
  27. Habitat-Matterport 3D semantics dataset. arXiv preprint arXiv:2210.05633, 2022.
  28. DeepPanoContext: Panoramic 3D scene understanding with holistic scene context graph and relation-based optimization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 12632–12641, 2021.
  29. PanoContext: A whole-room 3D context model for panoramic scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 668–686. Springer, 2014.
  30. Structured3D: A large photo-realistic dataset for structured 3D modeling. arXiv preprint arXiv:1908.00222, 2019.
  31. Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics (TOG), 37(4):1–12, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Qirui Wu (11 papers)
  2. Sonia Raychaudhuri (7 papers)
  3. Daniel Ritchie (50 papers)
  4. Manolis Savva (64 papers)
  5. Angel X Chang (2 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets