Papers
Topics
Authors
Recent
Search
2000 character limit reached

Context-Aware Indoor Point Cloud Object Generation through User Instructions

Published 26 Nov 2023 in cs.CV | (2311.16501v3)

Abstract: Indoor scene modification has emerged as a prominent area within computer vision, particularly for its applications in Augmented Reality (AR) and Virtual Reality (VR). Traditional methods often rely on pre-existing object databases and predetermined object positions, limiting their flexibility and adaptability to new scenarios. In response to this challenge, we present a novel end-to-end multi-modal deep neural network capable of generating point cloud objects seamlessly integrated with their surroundings, driven by textual instructions. Our model revolutionizes scene modification by enabling the creation of new environments with previously unseen object layouts, eliminating the need for pre-stored CAD models. Leveraging Point-E as our generative model, we introduce innovative techniques such as quantized position prediction and Top-K estimation to address the issue of false negatives resulting from ambiguous language descriptions. Furthermore, we conduct comprehensive evaluations to showcase the diversity of generated objects, the efficacy of textual instructions, and the quantitative metrics, affirming the realism and versatility of our model in generating indoor objects. To provide a holistic assessment, we incorporate visual grounding as an additional metric, ensuring the quality and coherence of the scenes produced by our model. Through these advancements, our approach not only advances the state-of-the-art in indoor scene modification but also lays the foundation for future innovations in immersive computing and digital environment creation.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Learning representations and generative models for 3D point clouds. In Int. Conf. Mach. Learn., pages 40–49, 2018.
  2. ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, 2020.
  3. Gaudi: A neural architect for immersive 3d scene generation. NeurIPS, 35:25102–25116, 2022.
  4. Language models are few-shot learners. In NeurIPS, pages 1877–1901, 2020.
  5. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
  6. Scanrefer: 3d object localization in rgb-d scans using natural language. ECCV, 2020.
  7. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian Conf. Comput. Vis., pages 100–116. Springer, 2019.
  8. End-to-end 3d dense captioning with vote2cap-detr. In CVPR, pages 11124–11133, 2023.
  9. Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, pages 3193–3203, 2021.
  10. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828–5839, 2017.
  11. Quickfps: Architecture and algorithm co-design for farthest point sampling in large-scale point clouds. IEEE Trans. Computer-Aided Des. Integr. Circuits and Syst., 2023.
  12. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  13. Denoising diffusion probabilistic models. NeurIPS, 33:6840–6851, 2020.
  14. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In ICCV, pages 7909–7920, 2023.
  15. Multi-view transformer for 3d visual grounding. In CVPR, pages 15524–15533, 2022.
  16. Progressive point cloud deconvolution generation network. In ECCV, pages 397–413. Springer, 2020.
  17. More: Multi-order relation mining for dense captioning in 3d scenes. In ECCV, pages 528–545. Springer, 2022.
  18. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. In CVPR, pages 10813–10823, 2023.
  19. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, 2019.
  20. Point cloud augmentation with weighted local transformations. In ICCV, pages 548–557, 2021.
  21. Point cloud generation using deep adversarial local features for augmented and mixed reality contents. IEEE Trans. Consum. Electron., 68(1):69–76, 2022.
  22. Towards implicit text-guided 3d shape generation. In CVPR, pages 17896–17906, 2022.
  23. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
  24. Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction. In CVPR, pages 12923–12932, 2023.
  25. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99–106, 2021.
  26. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
  27. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  28. Atiss: Autoregressive transformers for indoor scene synthesis. NeurIPS, 34:12013–12026, 2021.
  29. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 30, 2017.
  30. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS, 2022.
  31. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 8748–8763, 2021.
  32. Look outside the room: Synthesizing a consistent long-term 3d scene video from a single image. In CVPR, pages 3563–3573, 2022.
  33. Object insertion based data augmentation for semantic segmentation. In Int. Conf. Robot. and Automat., pages 359–365. IEEE, 2022.
  34. Fast and flexible indoor scene synthesis via deep convolutional generative models. In CVPR, pages 6182–6190, 2019.
  35. The earth mover’s distance as a metric for image retrieval. IJCV, 40:99–121, 2000.
  36. Deep unsupervised learning using nonequilibrium thermodynamics. In Int. Conf. Mach. Learn., pages 2256–2265, Lille, France, 2015.
  37. Attention is all you need. In NeurIPS, 2017.
  38. Sceneformer: Indoor scene generation with transformers. In 3DV, pages 106–115. IEEE, 2021.
  39. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In CVPR, pages 20908–20918, 2023.
  40. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, pages 4541–4550, 2019.
  41. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In ICCV, pages 1791–1800, 2021.
  42. Multi3drefer: Grounding text description to multiple 3d objects. In ICCV, pages 15225–15236, 2023.
  43. 3d shape generation and completion through point-voxel diffusion. In CVPR, pages 5826–5835, 2021.
  44. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In ICCV, pages 7384–7392, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up