OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields (2305.10503v3)

Abstract: The emergence of Neural Radiance Fields (NeRF) for novel view synthesis has increased interest in 3D scene editing. An essential task in editing is removing objects from a scene while ensuring visual reasonability and multiview consistency. However, current methods face challenges such as time-consuming object labeling, limited capability to remove specific targets, and compromised rendering quality after removal. This paper proposes a novel object-removing pipeline, named OR-NeRF, that can remove objects from 3D scenes with user-given points or text prompts on a single view, achieving better performance in less time than previous works. Our method spreads user annotations to all views through 3D geometry and sparse correspondence, ensuring 3D consistency with less processing burden. Then recent 2D segmentation model Segment-Anything (SAM) is applied to predict masks, and a 2D inpainting model is used to generate color supervision. Finally, our algorithm applies depth supervision and perceptual loss to maintain consistency in geometry and appearance after object removal. Experimental results demonstrate that our method achieves better editing quality with less time than previous works, considering both quality and quantity.