InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields (2305.15094v2)

Abstract: We propose InNeRF360, an automatic system that accurately removes text-specified objects from 360-degree Neural Radiance Fields (NeRF). The challenge is to effectively remove objects while inpainting perceptually consistent content for the missing regions, which is particularly demanding for existing NeRF models due to their implicit volumetric representation. Moreover, unbounded scenes are more prone to floater artifacts in the inpainted region than frontal-facing scenes, as the change of object appearance and background across views is more sensitive to inaccurate segmentations and inconsistent inpainting. With a trained NeRF and a text description, our method efficiently removes specified objects and inpaints visually consistent content without artifacts. We apply depth-space warping to enforce consistency across multiview text-encoded segmentations, and then refine the inpainted NeRF model using perceptual priors and 3D diffusion-based geometric priors to ensure visual plausibility. Through extensive experiments in segmentation and inpainting on 360-degree and frontal-facing NeRFs, we show that our approach is effective and enhances NeRF's editability. Project page: https://ivrl.github.io/InNeRF360.

• The paper presents a text-guided inpainting method that leverages depth-space warping to achieve consistent multiview segmentation in 360° NeRFs.
• It integrates 2D inpainting with 3D diffusion-based and contextual appearance priors to effectively remove artifacts such as floaters.
• Experimental results demonstrate superior performance over state-of-the-art techniques, enhancing visual coherence in immersive 3D scenes.

Overview of "black: Text-Guided 3D-Consistent Object Inpainting in 360-Degree Neural Radiance Fields"

The paper presents a novel system named "black," which addresses the task of inpainting within 360-degree Neural Radiance Fields (NeRF). The research focuses on removing specified objects from these scenes while filling in visually consistent content. The proposed method introduces a mechanism for handling the unique challenges associated with 360-degree NeRFs, such as artifact management and complex view dependencies.

Key Contributions and Methodology

The primary contribution of this work is introducing a text-guided inpainting method for NeRFs, focusing specifically on 360-degree scenes, which are notably more complex than front-facing scenes. The system utilizes depth-space warping to produce consistent multiview segmentations and leverages geometric and perceptual priors to achieve high-quality inpainting.

• Depth-Space Warping: The authors propose a depth-space warping mechanism to improve multiview segmentations. This method uses depth information to refine initial segmentation masks, ensuring that object identity is consistent across different viewing angles.
• NeRF Inpainting Pipeline: The inpainting process starts with rendering observations using the segmented masks. Using 2D inpainting methods as a base, the system fine-tunes the resultant NeRF using 3D diffusion-based priors to remove artifacts commonly known as "floaters" and applies contextual appearance priors to optimize textures.
• Geometric and Appearance Priors: To maintain scene consistency in the inpainted NeRF, the approach employs 3D geometric priors trained on extensive shape datasets and contextual appearance priors, enhancing robustness against inconsistencies in image-to-image variations.

Experimental Results

The paper reports comprehensive experiments on several datasets illustrating the system's efficacy in both segmentation and inpainting tasks within 360-degree scenes. Comparisons with state-of-the-art methods reveal that "black" achieves superior performance by reducing visual artifacts and enhancing consistency across viewpoints. Quantitative results measured using metrics such as LPIPS and FID reflect substantial improvements over methods like SPIn-NeRF, particularly in maintaining visual coherence without artifacts.

Implications and Future Directions

The findings in this paper have significant implications for the fields of virtual and augmented reality. By improving the controllability of NeRF representations, this research paves the way for more accessible and manipulatable 3D scene representations. For instance, the ability to remove and inpaint objects within a complete 360-degree context can significantly benefit applications in content creation and editing within immersive environments.

For future work, further exploration might focus on adapting the methodology to broader types of scenes and enhancing segmentation for more complex object configurations. Addressing limitations inherent in initial mask accuracy or in object detector models might also lead to improvements in precision and usability across diverse VR/AR applications.

In conclusion, "black" marks an important step in the evolution of NeRF technology, offering a framework for seamless object editing and enhanced scene pliability, which are crucial for further advancing 3D reconstruction and rendering capabilities.