VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2402.17427v1)
Abstract: Existing NeRF-based methods for large scene reconstruction often have limitations in visual quality and rendering speed. While the recent 3D Gaussian Splatting works well on small-scale and object-centric scenes, scaling it up to large scenes poses challenges due to limited video memory, long optimization time, and noticeable appearance variations. To address these challenges, we present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. We propose a progressive partitioning strategy to divide a large scene into multiple cells, where the training cameras and point cloud are properly distributed with an airspace-aware visibility criterion. These cells are merged into a complete scene after parallel optimization. We also introduce decoupled appearance modeling into the optimization process to reduce appearance variations in the rendered images. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets, enabling fast optimization and high-fidelity real-time rendering.
- Building rome in a day. Communications of the ACM, 2011.
- Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
- Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
- Zip-nerf: Anti-aliased grid-based neural radiance fields. In ICCV, 2023.
- Optimizing the latent space of generative networks. In ICML, 2018.
- Au-air: A multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance. In ICRA, 2020.
- Hexplane: A fast representation for dynamic scenes. In CVPR, 2023.
- Real-time neural light field on mobile devices. In CVPR, 2023.
- Tensorf: Tensorial radiance fields. In ECCV, 2022a.
- Hallucinated neural radiance fields in the wild. In CVPR, 2022b.
- Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In CVPR, 2023a.
- Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.16585, 2023b.
- The unmanned aerial vehicle benchmark: Object detection and tracking. In ECCV, 2018.
- Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
- K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
- An automated method for large-scale, ground-based city model acquisition. IJCV, 2004.
- Towards internet-scale multi-view stereo. In CVPR, 2010.
- Monocular dynamic view synthesis: A reality check. In NeurIPS, 2022.
- Multi-view stereo for community photo collections. In ICCV, 2007.
- Baking neural radiance fields for real-time view synthesis. In ICCV, 2021.
- 3d gaussian splatting for real-time radiance field rendering. ACM ToG, 2023.
- Aads: Augmented autonomous driving simulation using data-driven algorithms. Science robotics, 2019.
- Modeling and recognition of landmark image collections using iconic scene graphs. In ECCV, 2008.
- Neuralangelo: High-fidelity neural surface reconstruction. In CVPR, 2023.
- Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia, 2022a.
- Capturing, reconstructing, and simulating: the urbanscene3d dataset. In ECCV, 2022b.
- Robust dynamic radiance fields. In CVPR, 2023.
- Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713, 2023.
- Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
- Neural rerendering in the wild. In CVPR, 2019.
- Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
- Instant neural graphics primitives with a multiresolution hash encoding. ACM ToG, 2022.
- Neural scene graphs for dynamic scenes. In CVPR, 2021.
- Detailed real-time urban 3d reconstruction from video. IJCV, 2008.
- Ravi Ramamoorthi. Nerfs: The search for the best 3d representation. arXiv preprint arXiv:2308.02751, 2023.
- Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, 2021.
- Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM ToG, 2023.
- Structure-from-motion revisited. In CVPR, 2016.
- Photo tourism: exploring photo collections in 3d. In SIGGRAPH. 2006.
- Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
- Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
- Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023a.
- Delicate textured mesh recovery from nerf via adaptive surface refinement. In ICCV, 2023b.
- Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022.
- Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022.
- R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis. In ECCV, 2022.
- Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
- F2-nerf: Fast neural radiance field training with free camera trajectories. In CVPR, 2023a.
- Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In ICCV, 2023b.
- Humannerf: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
- 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
- Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, 2022.
- Grid-guided neural radiance fields for large urban scenes. In CVPR, 2023.
- Surfelgan: Synthesizing realistic sensor data for autonomous driving. In CVPR, 2020.
- Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101, 2023a.
- Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642, 2023b.
- Volume rendering of neural implicit surfaces. In NeurIPS, 2021.
- Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859, 2023.
- Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529, 2023.
- Plenoctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
- Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In ICLR, 2022.
- Very large-scale global sfm by distributed motion averaging. In CVPR, 2018.
- Ewa volume splatting. In VIS, 2001.