Urban Radiance Field Representation with Deformable Neural Mesh Primitives (2307.10776v1)

Abstract: Neural Radiance Fields (NeRFs) have achieved great success in the past few years. However, most current methods still require intensive resources due to ray marching-based rendering. To construct urban-level radiance fields efficiently, we design Deformable Neural Mesh Primitive~(DNMP), and propose to parameterize the entire scene with such primitives. The DNMP is a flexible and compact neural variant of classic mesh representation, which enjoys both the efficiency of rasterization-based rendering and the powerful neural representation capability for photo-realistic image synthesis. Specifically, a DNMP consists of a set of connected deformable mesh vertices with paired vertex features to parameterize the geometry and radiance information of a local area. To constrain the degree of freedom for optimization and lower the storage budgets, we enforce the shape of each primitive to be decoded from a relatively low-dimensional latent space. The rendering colors are decoded from the vertex features (interpolated with rasterization) by a view-dependent MLP. The DNMP provides a new paradigm for urban-level scene representation with appealing properties: $(1)$ High-quality rendering. Our method achieves leading performance for novel view synthesis in urban scenarios. $(2)$ Low computational costs. Our representation enables fast rendering (2.07ms/1k pixels) and low peak memory usage (110MB/1k pixels). We also present a lightweight version that can run 33$\times$ faster than vanilla NeRFs, and comparable to the highly-optimized Instant-NGP (0.61 vs 0.71ms/1k pixels). Project page: \href{https://dnmp.github.io/}{https://dnmp.github.io/}.

• The paper introduces DNMPs that blend neural networks with traditional mesh rendering to efficiently synthesize photorealistic urban scenes.
• The method leverages hierarchical voxelization and low-dimensional latent spaces to reduce computational costs and reliably handle incomplete geometry.
• Experiments on KITTI-360 and Waymo datasets demonstrate faster rendering (2.07ms per 1k pixels) and competitive image quality against established benchmarks.

Urban Radiance Field Representation with Deformable Neural Mesh Primitives

The paper "Urban Radiance Field Representation with Deformable Neural Mesh Primitives" presents a novel approach to synthesizing photo-realistic images in urban scenes using a technique that leverages a blend of traditional mesh-based rendering efficiency and the expressive power of neural networks. This work introduces the concept of Deformable Neural Mesh Primitives (DNMPs) as a representation scheme for neural radiance fields, offering improvements in rendering speed and accuracy over previous methods.

Among the significant advancements highlighted in this paper is the DNMP which serves as a neural extension to classical mesh representations. By embedding the rich features of traditional meshes within a neural framework, the authors achieve compactness and efficiency in rasterization-based rendering and a high capacity for photorealistic image synthesis. A DNMP consists of deformable mesh vertices linked to vertex features that encapsulate local geometric and radiance information. An essential aspect of this methodology is the constraint on the degrees of freedom achieved by decoding the shapes from a low-dimensional latent space, allowing the rendering to remain both efficient and robust to practical scenarios, such as urban outdoor environments.

A core contribution of the paper is addressing the efficiency concerns inherent in neural radiance fields, particularly regarding ray-marching techniques that can be computationally expensive. The DNMP framework circumvents this by enabling fast rasterization and efficient scene representation through a hierarchical voxelization approach, significantly reducing the computational resources needed and avoiding unnecessary sampling of empty spaces. The introduction of hierarchical DNMPs allows the model to cover areas with incomplete depth information effectively, enhancing the robustness of the reconstruction in urban settings.

The paper provides evidence of performance improvements through extensive experiments on urban datasets like KITTI-360 and the Waymo Open Dataset. The proposed method produced not only faster rendering times—achieving 2.07ms per 1k pixels with 110MB peak memory usage—but also maintained high-quality synthesis outperforming established benchmarks in key metrics such as PSNR, SSIM, and LPIPS. Particularly notable is the lightweight version of the approach that outpaces the highly-optimized Instant-NGP, rendering at comparable speeds (0.61 versus 0.71ms per 1k pixels) while maintaining competitive visual fidelity.

Furthermore, the method extends the practicality of scene manipulation into applications such as VR/AR, facilitated by the inherent mesh-based structure that supports editing tasks like texture modification and object manipulation with minimal computational overhead. The paper also addresses potential future extensions of the research into dynamic scenes, suggesting avenues for subfield development.

In conclusion, this work introduces a novel and efficient paradigm for neural rendering in urban environments that could serve as a foundation for further exploration in the domain of neural graphics. By effectively bridging the gap between mesh-based explicit geometry and neural implicit functions, the paper contributes a meaningful step forward in the development of scalable, realistic rendering techniques suitable for complex real-world applications.