Papers
Topics
Authors
Recent
2000 character limit reached

SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting (2312.15856v3)

Published 26 Dec 2023 in cs.GR and cs.CV

Abstract: A key challenge in fine-grained 3D-based interactive editing is the absence of an efficient representation that balances diverse modifications with high-quality view synthesis under a given memory constraint. While 3D meshes provide robustness for various modifications, they often yield lower-quality view synthesis compared to 3D Gaussian Splatting, which, in turn, suffers from instability during extensive editing. A straightforward combination of these two representations results in suboptimal performance and fails to meet memory constraints. In this paper, we introduce SplatMesh, a novel fine-grained interactive 3D segmentation and editing algorithm that integrates 3D Gaussian Splat with a precomputed mesh and could adjust the memory request based on the requirement. Specifically, given a mesh, \method simplifies it while considering both color and shape, ensuring it meets memory constraints. Then, SplatMesh aligns Gaussian splats with the simplified mesh by treating each triangle as a new reference point. By segmenting and editing the simplified mesh, we can effectively edit the Gaussian splats as well, which will lead to extensive experiments on real and synthetic datasets, coupled with illustrative visual examples, highlighting the superiority of our approach in terms of representation quality and editing performance. Code of our paper can be found here: https://github.com/kaichen-z/SplatMesh.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Neural point-based graphics. 2020.
  2. Signed distance computation using the angle weighted pseudonormal. IEEE Transactions on Visualization and Computer Graphics, 11(3):243–253, 2005.
  3. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20919–20929, 2023.
  4. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
  5. Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227, 2022.
  6. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  7. Segment anything in 3d with nerfs. arXiv preprint arXiv:2304.12308, 2023.
  8. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124–14133, 2021.
  9. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022a.
  10. UPST-NeRF: Universal photorealistic style transfer of neural radiance fields for 3d scene. In arXiv preprint arXiv:2208.07059, 2022b.
  11. Stylizing 3d scene via implicit representation and hypernetwork. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1475–1484, 2022.
  12. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In European Conference on Computer Vision (ECCV), 2022.
  13. Neural point cloud rendering via multi-plane projection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7830–7839, 2020.
  14. Unified implicit neural stylization. arXiv preprint arXiv:2204.01943, 2022a.
  15. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776, 2022b.
  16. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 3DV, 2022.
  17. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems, 2022.
  18. Interactive segmentation of radiance fields. arXiv preprint arXiv:2212.13545, 2022.
  19. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503, 2020.
  20. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03380, 2022.
  21. Instance neural radiance field. arXiv preprint arXiv:2304.04395, 2023a.
  22. Deep real-time volumetric rendering using multi-feature fusion. In ACM SIGGRAPH 2023 Conference Proceedings, New York, NY, USA, 2023b. Association for Computing Machinery.
  23. Stylizednerf: consistent 3d scene stylization as stylized nerf via 2d-3d mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18342–18352, 2022.
  24. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406–413, 2014.
  25. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553, 2023.
  26. 3d object manipulation in a single photograph using stock 3d models. ACM Transactions on Computer Graphics, 33(4), 2014.
  27. Decomposing nerf for editing via feature field distillation. In NeurIPS, 2022.
  28. Palettenerf: Palette-based appearance editing of neural radiance fields. arXiv preprint arXiv:2212.10699, 2022.
  29. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020a.
  30. Crowdsampling the plenoptic function. In European Conference on Computer Vision, pages 178–196. Springer, 2020b.
  31. Physically-based editing of indoor scene lighting from a single image. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI, pages 555–572. Springer, 2022.
  32. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
  33. Editing conditional radiance fields. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  34. Unsupervised multi-view object segmentation using radiance field propagation. In NeurIPS, 2022.
  35. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 38(4):65:1–65:14, 2019.
  36. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019a.
  37. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1–14, 2019b.
  38. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020a.
  39. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020b.
  40. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022.
  41. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280–8290, 2022.
  42. GIRAFFE: representing scenes as compositional generative neural feature fields. In CVPR, 2021.
  43. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
  44. Poisson image editing. In ACM SIGGRAPH 2003 Papers, pages 313–318. 2003.
  45. Vision transformers for dense prediction. ArXiv preprint, 2021.
  46. Neural volumetric object selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6133–6142, 2022a.
  47. Neural volumetric object selection. In CVPR, 2022b.
  48. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  49. ”grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 2004.
  50. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
  51. Adversarial scene editing: Automatic object removal from weak supervision. In Advances in Neural Information Processing Systems 31, pages 7716–7726, Montréal, Canada, 2018. Curran Associates.
  52. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  53. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2019a.
  54. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019b.
  55. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
  56. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
  57. Compressible-composable nerf via rank-residual decomposition. arXiv preprint arXiv:2205.14870, 2022.
  58. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In 3DV, 2022.
  59. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXiv:2111.13260, 2021.
  60. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021.
  61. Dual octree graph networks for learning adaptive volumetric shape representations. ACM Transactions on Graphics (TOG), 41(4):1–15, 2022.
  62. Seal-3d: Interactive pixel-level editing for neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17683–17693, 2023.
  63. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210–16220, 2022.
  64. Neutex: Neural texture mapping for volumetric neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7119–7128, 2021.
  65. Learning object-compositional neural radiance field for editable scene rendering. In International Conference on Computer Vision (ICCV), 2021.
  66. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492–2502, 2020.
  67. Volume rendering of neural implicit surfaces. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  68. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis. 2022.
  69. Unsupervised discovery of object radiance fields. In ICLR, 2022a.
  70. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022b.
  71. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18353–18364, 2022.
  72. Arf: Artistic radiance fields. In European Conference on Computer Vision, pages 717–733. Springer, 2022.
  73. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1–18, 2021.
  74. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
  75. Dynpoint: Dynamic neural point for view synthesis. arXiv preprint arXiv:2310.18999, 2023.
  76. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
  77. Learning-based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing. In SIGGRAPH Asia 2022 Conference Papers. ACM, 2022.
  78. I2-sdf: Intrinsic indoor scene reconstruction and editing via raytracing in neural sdfs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12489–12498, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 tweet and received 3 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial