VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors (2407.02945v3)

Abstract: Neural rendering-based urban scene reconstruction methods commonly rely on images collected from driving vehicles with cameras facing and moving forward. Although these methods can successfully synthesize from views similar to training camera trajectory, directing the novel view outside the training camera distribution does not guarantee on-par performance. In this paper, we tackle the Extrapolated View Synthesis (EVS) problem by evaluating the reconstructions on views such as looking left, right or downwards with respect to training camera distributions. To improve rendering quality for EVS, we initialize our model by constructing dense LiDAR map, and propose to leverage prior scene knowledge such as surface normal estimator and large-scale diffusion model. Qualitative and quantitative comparisons demonstrate the effectiveness of our methods on EVS. To the best of our knowledge, we are the first to address the EVS problem in urban scene reconstruction. Link to our project page: https://vegs3d.github.io/.

• The paper introduces an urban scene reconstruction method leveraging LiDAR initialization, surface normal guidance, and diffusion model distillation to improve view extrapolation.
• It employs dynamic scene modeling with 3D Gaussian Splatting to separate static and dynamic components and address covariance optimization challenges.
• Experimental results on KITTI datasets demonstrate superior visual quality and metrics, outperforming state-of-the-art methods like Mip-NeRF 360 and BlockNeRF++.

View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors

The research paper "View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors" presents an exploration into expanding the capabilities of neural rendering-based urban scene reconstruction by addressing the challenge of extrapolated view synthesis (EVS). This involves generating high-quality images from camera positions and orientations not encountered during training. Using the context of urban scenes, traditionally captured by forward-facing cameras mounted on vehicles, the paper systematically develops methods to overcome the limitations associated with such constrained viewpoints.

Methodology

The authors introduce the concept of initializing a neural model with a dense LiDAR map of the scene, leveraging multiple priors, including surface normal estimators and large-scale diffusion models, to enhance the rendering quality of extrapolated views. The core approach centers around three main contributions:

1. Dynamic Scene Modeling and Initialization: The authors propose constructing a dense point cloud map from LiDAR data and integrating it into a dynamic scene model. This model separates static and dynamic components, initializing Gaussian means with dense LiDAR points. Methods like 3D Gaussian Splatting are employed to represent the scene with point-based models, enabling real-time high-fidelity rendering.
2. Covariance Guidance with Surface Normal Prior: A novel technique is proposed where covariance matrices of the Gaussians are guided by surface normals to ensure their shapes and orientations closely adhere to the underlying scene geometry. This mitigates the 'lazy covariance optimization problem', ensuring that the rendered covariances do not exhibit distorted cavities when viewed from extrapolated angles. The paper introduces the covariance axis loss and covariance scale loss to align covariance orientations and scales with predicted surface normals effectively.
3. Visual Knowledge Distillation from Large-Scale Diffusion Model: For direct supervision of extrapolated views, knowledge from a fine-tuned image diffusion model (Stable Diffusion) is distilled. This model is adapted using LoRA (Low-Rank Adaptation) to balance generalization and scene-specific visual fidelity. The concept of denoising score matching is employed, wherein noise prediction from the diffusion model is used to approximate and adjust the rendered images towards more plausible visual representations.

Experimental Results

The method was evaluated on the KITTI and KITTI-360 datasets, focusing on newly defined EVS scenarios where cameras look left, right, and downwards significantly from training camera trajectories. Quantitative comparisons included metrics such as FID, KID, PSNR, SSIM, and LPIPS across various views. The authors demonstrated notable improvements in rendering quality in EVS conditions over state-of-the-art methods like Mip-NeRF 360, BlockNeRF++, MARS, and the baseline 3D Gaussian Splatting method.

Implications and Future Directions

The contributions of this research have substantial implications for the field of neural rendering and scene reconstruction, particularly in dynamic urban environments. By improving the quality of extrapolated views, applications in autonomous driving, urban planning, and virtual reality can achieve higher visual fidelity and accuracy. The effective integration of LiDAR data, surface normal priors, and diffusion models underscores the potential for multi-sensory data fusion in enhancing neural rendering frameworks.

The research opens several avenues for future work. Extending the approach to handle even more diverse urban environments with varying lighting and atmospheric conditions is a natural progression. Additionally, exploring the integration of other sensory inputs, such as thermal imagery or radar, could provide further robustness and accuracy. Finally, advancements in reducing the computational overhead and optimizing training times remain critical for deploying these methods in real-time applications.

In conclusion, the authors of this paper have adeptly tackled a challenging aspect of urban scene reconstruction, enhancing neural rendering techniques for views beyond conventional camera trajectories. Their innovative use of scene priors and diffusion model fine-tuning sets a new benchmark in the field, promising to significantly impact practical applications ranging from autonomous navigation to immersive virtual experiences.