Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes (2401.05335v1)

Published 10 Jan 2024 in cs.CV, cs.GR, and cs.LG

Abstract: We introduce InseRF, a novel method for generative object insertion in the NeRF reconstructions of 3D scenes. Based on a user-provided textual description and a 2D bounding box in a reference viewpoint, InseRF generates new objects in 3D scenes. Recently, methods for 3D scene editing have been profoundly transformed, owing to the use of strong priors of text-to-image diffusion models in 3D generative modeling. Existing methods are mostly effective in editing 3D scenes via style and appearance changes or removing existing objects. Generating new objects, however, remains a challenge for such methods, which we address in this study. Specifically, we propose grounding the 3D object insertion to a 2D object insertion in a reference view of the scene. The 2D edit is then lifted to 3D using a single-view object reconstruction method. The reconstructed object is then inserted into the scene, guided by the priors of monocular depth estimation methods. We evaluate our method on various 3D scenes and provide an in-depth analysis of the proposed components. Our experiments with generative insertion of objects in several 3D scenes indicate the effectiveness of our method compared to the existing methods. InseRF is capable of controllable and 3D-consistent object insertion without requiring explicit 3D information as input. Please visit our project page at https://mohamad-shahbazi.github.io/inserf.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18208–18218, 2022.
  2. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  3. Gaudi: A neural architect for immersive 3d scene generation. arXiv, 2022.
  4. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
  5. Efficient geometry-aware 3D generative adversarial networks. In arXiv, 2021.
  6. Dense pixel-to-pixel harmonization via continuous image representation. IEEE Transactions on Circuits and Systems for Video Technology, pages 1–1, 2023a.
  7. Zero-shot image harmonization with generative model prior, 2023b.
  8. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142–13153, 2023.
  9. ViCA-neRF: View-consistency-aware 3d editing of neural radiance fields. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  10. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  11. Prompt-to-prompt image editing with cross attention control. In arXiv preprint arXiv:2208.01626, 2022.
  12. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239, 2020.
  13. Focaldreamer: Text-driven 3d editing via focal-fusion assembly, 2023.
  14. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  15. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization, 2023a.
  16. Zero-1-to-3: Zero-shot one image to 3d object, 2023b.
  17. Syncdreamer: Learning to generate multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023c.
  18. Wonder3d: Single image to 3d using cross-domain diffusion, 2023.
  19. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11461–11471, 2022.
  20. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  21. Reference-guided controllable inpainting of neural radiance fields. In ICCV, 2023a.
  22. Watch your steps: Local image and scene editing by text instructions. In arXiv, 2023b.
  23. SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, 2023c.
  24. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  25. Autodecoding latent 3d diffusion models, 2023.
  26. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf, 2023.
  27. Localizing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  28. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
  29. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843, 2023.
  30. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  31. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
  32. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2305.16213, 2023.
  33. High-resolution image synthesis with latent diffusion models, 2021.
  34. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In arXiv preprint arxiv:2208.12242, 2022.
  35. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
  36. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  37. Language-driven object fusion into neural radiance fields with pose-conditioned dataset updates, 2023.
  38. Blending-nerf: Text-driven localized editing in neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14383–14393, 2023.
  39. Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv preprint arXiv:2305.15094, 2023a.
  40. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213, 2023b.
  41. Removing objects from neural radiance fields. In CVPR, 2023.
  42. Or-nerf: Object removing from 3d scenes guided by multiview segmentation with neural radiance fields, 2023.
  43. Edit-diffnerf: Editing 3d neural radiance fields using 2d diffusion model, 2023.
  44. Magicbrush: A manually annotated dataset for instruction-guided image editing. In Advances in Neural Information Processing Systems, 2023a.
  45. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.09618, 2023b.
  46. Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint arXiv:2306.13455, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Mohamad Shahbazi (9 papers)
  2. Liesbeth Claessens (1 paper)
  3. Michael Niemeyer (29 papers)
  4. Edo Collins (5 papers)
  5. Alessio Tonioni (32 papers)
  6. Luc Van Gool (570 papers)
  7. Federico Tombari (214 papers)
Citations (5)
View on Semantic Scholar

Summary

Introduction to InseRF

Within the field of 3D scene generation and manipulation, a significant innovation has been made with a method designed for the generative insertion of objects into 3D scenes using textual descriptions. The technology, known aptly as InseRF, overcomes existing hurdles of generating new objects within a scene with remarkable consistency and realism.

The Approach

InseRF is distinct in its approach. It begins with a 2D edit of the object based on text and a bounding box in a reference view. This image is then transformed into a 3D model using a single-view reconstruction mechanism. Significantly, this process does not demand explicit 3D coordinates for placement, relying rather on monocular depth estimation methods to guide the object's insertion into the scene.

To further strengthen the inserted object's fidelity, InseRF includes an optional refinement phase, optimizing the newly introduced object's scale, rotation, and translation to better meld it with the existing environment, enhancing the realism of the insertion.

Experimental Findings

The evaluation of InseRF involved testing on various real-world indoor and outdoor scenes. The results underscored InseRF's capacity to insert objects into complex scenes with precision, purely steered by a textual description without explicit spatial 3D information. Visual comparisons underline the superiority of InseRF against other methods, with the ability to seamlessly integrate the new objects within the scene's fabric.

Looking Ahead

Despite substantial strides made by InseRF, its performance aligns closely with the capabilities of underlying generative models. Improvements in these arenas will directly augment InseRF's efficacy. Moreover, incorporating advancements in scene-consistent shadowing and harmonization could further polish the authenticity of the inserted objects.

InseRF ushers in the future of editable 3D environments, where the limits of alteration and creation are expanded exponentially, offering vast new potentials in digital content creation and virtual environment design.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes