Grid-guided Neural Radiance Fields for Large Urban Scenes (2303.14001v1)

Abstract: Purely MLP-based neural radiance fields (NeRF-based methods) often suffer from underfitting with blurred renderings on large-scale scenes due to limited model capacity. Recent approaches propose to geographically divide the scene and adopt multiple sub-NeRFs to model each region individually, leading to linear scale-up in training costs and the number of sub-NeRFs as the scene expands. An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions. However, the feature grid tends to be less constrained and often reaches suboptimal solutions, producing noisy artifacts in renderings, especially in regions with complex geometry and texture. In this work, we present a new framework that realizes high-fidelity rendering on large urban scenes while being computationally efficient. We propose to use a compact multiresolution ground feature plane representation to coarsely capture the scene, and complement it with positional encoding inputs through another NeRF branch for rendering in a joint learning fashion. We show that such an integration can utilize the advantages of two alternative solutions: a light-weighted NeRF is sufficient, under the guidance of the feature grid representation, to render photorealistic novel views with fine details; and the jointly optimized ground feature planes, can meanwhile gain further refinements, forming a more accurate and compact feature space and output much more natural rendering results.

• The paper presents a two-branch model that combines grid-based feature representations with NeRF for capturing both coarse structure and fine details.
• The multi-resolution feature grid significantly reduces computational complexity from O(N^3) to O(N^2) while preserving scene quality.
• Joint optimization of grid and NeRF branches leads to enhanced photorealism and improved metrics such as PSNR, SSIM, and LPIPS.

Grid-Guided Neural Radiance Fields for Large Urban Scenes

The paper "Grid-guided Neural Radiance Fields for Large Urban Scenes" presents an innovative approach for modeling expansive urban environments using a combination of neural radiance fields and feature grid representations. The core contribution of this research is a framework that integrates NeRF-based methods, known for their photorealistic rendering capabilities and scene continuity, with feature grid-based methods that offer computational efficiency and scalability. By leveraging both approaches, the paper proposes a method capable of achieving photorealistic novel view renderings in large urban scenes, overcoming common limitations such as underfitting and noisy artifacts.

Key Contributions

1. Two-Branch Model Architecture: The proposed framework utilizes a two-branch model that combines NeRF-based and grid-based methodologies. The grid branch captures the coarse geometry of the scene using a multi-resolution feature plane representation, while the NeRF branch, guided by the feature grid, focuses on rendering fine details.
2. Multi-Resolution Feature Grid Representation: The paper introduces a compact factorization method to approximate full 3D feature grids without loss of quality. The ground feature planes that span the scene are coupled with a vertically shared feature vector along the z-axis. This reduces the computational burden significantly from $\mathcal{O}(N^3)$ to $\mathcal{O}(N^2)$.
3. Refinement Via Joint Training: The feature grids are jointly optimized with the NeRF branch in the second stage of training. This approach utilizes the provided global continuity from NeRF's MLP architecture to mitigate local suboptimal solutions and produces more refined grid features.

Experimental Evaluation

The paper thoroughly evaluates the proposed method against existing NeRF-based models (such as NeRF and Mega-NeRF) and grid-based models (such as TensoRF) using large urban scene datasets. Results demonstrate superior performance with notable improvements in PSNR, SSIM, and LPIPS metrics, as well as qualitative assessments. The proposed framework shows significant advancements in capturing sharp geometric features, producing consistent textures, and achieving smoother spatial continuity in novel view renderings.

Implications and Future Directions

The findings of this paper have practical implications for applications such as autonomous vehicle simulations, aerial surveying, and virtual reality, where large-scale scene modeling is essential. The integration of NeRF and grid-based methods could pave the way for more efficient and scalable scene modeling solutions. Future research could explore further optimizations in neural architecture and training strategies, as well as extending this framework to accommodate dynamic scenes and real-time rendering scenarios. Additionally, adapting this methodology for scenes with differing complexities and scales remains a promising direction for further paper.

In conclusion, the paper provides a significant advancement in the neural rendering domain by effectively addressing the challenges associated with large urban scene modeling using a novel grid-guided neural radiance fields approach.