CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians (2404.01133v3)

Abstract: The advancement of real-time 3D scene reconstruction and novel view synthesis has been significantly propelled by 3D Gaussian Splatting (3DGS). However, effectively training large-scale 3DGS and rendering it in real-time across various scales remains challenging. This paper introduces CityGaussian (CityGS), which employs a novel divide-and-conquer training approach and Level-of-Detail (LoD) strategy for efficient large-scale 3DGS training and rendering. Specifically, the global scene prior and adaptive training data selection enables efficient training and seamless fusion. Based on fused Gaussian primitives, we generate different detail levels through compression, and realize fast rendering across various scales through the proposed block-wise detail levels selection and aggregation strategy. Extensive experimental results on large-scale scenes demonstrate that our approach attains state-of-theart rendering quality, enabling consistent real-time rendering of largescale scenes across vastly different scales. Our project page is available at https://dekuliutesla.github.io/citygs/.

• The paper presents a novel divide-and-conquer strategy that trains large-scale 3D Gaussian Splatting models in parallel to reduce memory overheads.
• It introduces a Level-of-Detail scheme with a global scene prior to ensure seamless, multi-scale rendering for real-time applications.
• CityGaussian achieves superior rendering metrics, significantly improving SSIM, PSNR, and LPIPS compared to state-of-the-art methods.

Real-time High-quality Large-scale Scene Rendering with CityGaussian

Introduction

The paper introduces an innovative approach titled CityGaussian (CityGS), aimed at significantly enhancing the rendering fidelity for large-scale scenes such as cities, combined with the efficiency required for real-time performance across varying scales. The method is predicated on the utilization of a novel divide-and-conquer strategy for effective large-scale 3D Gaussian Splatting (3DGS) training, and a Level-of-Detail (LoD) strategy that ensures seamless and fast rendering across various scales. This approach effectively addresses challenges associated with training and memory overheads in large-scale scene reconstruction, paving the way for consistent real-time rendering without compromising visual quality.

Technical Contribution

CityGS endeavors to remedy the limitations of existing 3D Gaussian Splatting techniques when applied to large-scale scenes. The paper's contributions are articulated through a series of methodological innovations and technical advancements:

• Introduction of a Divide-and-Conquer Strategy: The paper presents a divide-and-conquer approach for the parallel training of large-scale 3D Gaussian Splatting, efficiently managing GPU memory restrictions and computational burdens by partitioning the global scene into blocks for focused training.
• Efficient Level-of-Detail Strategy: A novel LoD strategy is proposed, leveraging different detail levels of Gaussians to maintain rendering fidelity while substantially reducing the GPU memory footprint and computation times, ensuring real-time performance across various viewing scales.
• Implementation of a Global Scene Prior: The strategy incorporates a global scene prior, derived from a coarse training phase, ensuring the alignment and seamlessness of blocks in the final rendered scene, mitigating discontinuities at block edges, and enhancing overall visual fidelity.

Empirical Evaluation

The efficacy of CityGS is substantiated through extensive experiments conducted on large-scale scenes, demonstrating superior rendering quality and real-time performance. The experiments reveal that:

• CityGS significantly outperforms state-of-the-art methods in rendering quality metrics (SSIM, PSNR, and LPIPS) across several benchmark datasets.
• The proposed LoD strategy enables the method to maintain high rendering speeds, achieving consistent real-time performance even under extreme varying scales, a feat not possible with previous techniques.

Implications and Future Directions

The research offers significant theoretical and practical implications for the field of large-scale scene rendering:

• Enhanced Scene Fidelity and Efficiency: The method sets a new benchmark in rendering large-scale scenes with high fidelity at real-time speeds, bridging the gap between visual quality and performance requirements for applications in VR/AR, autonomous driving, and urban planning.
• Foundation for Future Research: By effectively addressing the limitations of existing 3DGS techniques in large-scale applications, this work lays a foundational framework for further exploration into efficient scene rendering techniques, posing potential shifts in how detailed virtual worlds are constructed and interacted with.
• Potential for Interactive Scene Manipulation: Given its explicit representation model, CityGS opens avenues for interactive manipulation of large-scale scenes, a capability that could revolutionize content creation in digital environments and simulation scenarios.

Conclusion

In conclusion, CityGaussian represents a significant advancement in the rendering of large-scale scenes, providing an effective solution to the challenges of training scalability and rendering performance. Its innovative divide-and-conquer training approach, coupled with the strategic implementation of Level-of-Detail, ushers in a new era of efficient, high-fidelity scene rendering suitable for real-time applications. This work not only demonstrates outstanding performance in current benchmarks but also sets the stage for future explorations into the efficient rendering of expansive virtual worlds.