Point'n Move: Interactive Scene Object Manipulation on Gaussian Splatting Radiance Fields (2311.16737v1)
Abstract: We propose Point'n Move, a method that achieves interactive scene object manipulation with exposed region inpainting. Interactivity here further comes from intuitive object selection and real-time editing. To achieve this, we adopt Gaussian Splatting Radiance Field as the scene representation and fully leverage its explicit nature and speed advantage. Its explicit representation formulation allows us to devise a 2D prompt points to 3D mask dual-stage self-prompting segmentation algorithm, perform mask refinement and merging, minimize change as well as provide good initialization for scene inpainting and perform editing in real-time without per-editing training, all leads to superior quality and performance. We test our method by performing editing on both forward-facing and 360 scenes. We also compare our method against existing scene object removal methods, showing superior quality despite being more capable and having a speed advantage.
- Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
- Segment anything in 3d with nerfs. arXiv:2304.12308, 2023.
- NeuralEditor: Editing neural radiance fields via manipulating point clouds. In CVPR, 2023.
- Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In European Conference on Computer Vision (ECCV), 2022.
- GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6629–6640, 2017.
- 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
- Segment Anything, 2023.
- Control-nerf: Editable feature volumes for scene rendering and manipulation. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4329–4339, 2022.
- Editing conditional radiance fields, 2021.
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pages 405–421, 2020.
- Reference-guided controllable inpainting of neural radiance fields. In ICCV, 2023a.
- SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, 2023b.
- Differentiable blocks world: Qualitative 3d decomposition by rendering primitives. ArXiv, abs/2307.05473, 2023.
- CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation. In Advances in Neural Information Processing Systems, pages 31402–31415, 2022.
- Sharf: Shape-conditioned Radiance Fields from a Single View. In Proceedings of the 38th International Conference on Machine Learning, pages 8948–8958, 2021.
- Resolution-robust Large Mask Inpainting with Fourier Convolutions. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3172–3182, 2022.
- Rip-nerf: Learning rotation-invariant point-based neural radiance field for fine-grained editing and compositing. Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, 2023.
- Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13:600–612, 2004.
- Removing objects from neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16528–16538, 2023.
- Deforming Radiance Fields with Cages. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 159–175, 2022.
- Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13759–13768, 2021.
- Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG), 38(6):1–14, 2019.
- Or-nerf: Object removing from 3d scenes guided by multiview segmentation with neural radiance fields, 2023.
- NeRF-Editing: Geometry Editing of Neural Radiance Fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18332–18343, 2022.
- The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
- Papr: Proximity attention point rendering. Advances in Neural Information Processing Systems, 2023.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.