MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2302.12249v1)

Abstract: Neural radiance fields enable state-of-the-art photorealistic view synthesis. However, existing radiance field representations are either too compute-intensive for real-time rendering or require too much memory to scale to large scenes. We present a Memory-Efficient Radiance Field (MERF) representation that achieves real-time rendering of large-scale scenes in a browser. MERF reduces the memory consumption of prior sparse volumetric radiance fields using a combination of a sparse feature grid and high-resolution 2D feature planes. To support large-scale unbounded scenes, we introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection. We design a lossless procedure for baking the parameterization used during training into a model that achieves real-time rendering while still preserving the photorealistic view synthesis quality of a volumetric radiance field.

• The paper introduces MERF, which reduces memory consumption and enables real-time rendering using a hybrid sparse 3D grid and high-resolution 2D feature planes.
• It employs a novel piecewise-projective contraction function to efficiently map scene coordinates, simplifying ray-box intersection computations in unbounded spaces.
• Experimental results demonstrate that MERF outperforms state-of-the-art methods with higher PSNR and SSIM while operating on memory-constrained devices.

Memory-Efficient Radiance Fields for Real-time View Synthesis

The paper "MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes" addresses a significant challenge in computer graphics, particularly in rendering high-quality images of large-scale scenes in real-time. Traditional neural radiance field (NeRF) representations, known for their photorealistic rendering capabilities, struggle with real-time performance due to computational intensity and memory constraints when dealing with large environments. This paper introduces Memory-Efficient Radiance Field (MERF), which proposes an innovative approach to tackle these constraints while preserving rendering quality.

Core Contributions and Methodology

The MERF framework is designed to reduce memory consumption and enhance the efficiency of real-time rendering for large-scale scenes. It achieves this through the following key innovations:

1. Sparse Feature Grid and High-resolution 2D Feature Planes: MERF uses a combination of a low-resolution 3D voxel grid and high-resolution 2D feature planes. The 3D grid provides a sparse volumetric structure, while the 2D planes capture detailed features, effectively balancing memory usage and image quality.
2. Novel Contraction Function: The paper introduces a piecewise-projective contraction function that maps scene coordinates into a bounded volume. This mapping maintains efficiency in ray-box intersection computations, which is crucial for addressing unbounded scenes and enables effective space skipping, a technique essential for accelerating the rendering process.
3. Optimization and Baking Strategy: MERF employs an initial optimization phase using a compressed hash grid structure, which allows for efficient training and subsequently bakes this optimized model into a format conducive to real-time rendering. The process preserves the photorealistic quality by ensuring that the baking step does not degrade the representation fidelity.

The method includes a uniquely defined parameterization of the scene, differentiating between density, diffuse color, and view-dependent features, which are mapped through nonlinear functions for an enhanced represenational power without excessively increasing computational cost.

Results and Implications

The experimental results from testing MERF demonstrate significant improvements in rendering speed and quality while drastically reducing memory requirements when compared to existing methods. MERF is shown to outperform real-time view synthesis methods like Mobile-NeRF by achieving higher image quality for the same level of compression and notably exceeds the performance of methods that require vastly more memory resources.

Key numerical results include:

• Enhanced PSNR and SSIM scores compared to contemporaneous real-time methods.
• The ability to scale efficiently with the memory footprint remaining manageable, making it viable for web-based applications.

Future Directions and Practical Impact

The introduction of MERF sets a new precedent for real-time rendering in computer graphics, especially applicable in scenarios demanding flexibility such as virtual reality or interactive web applications. Its efficiency and memory-conscious design highlight a step towards deploying complex 3D rendering capabilities on less powerful consumer devices, potentially democratizing access to high-quality immersive experiences.

Theoretically, the techniques involving contraction functions and hybrid parameterization structures proposed in this paper could further influence the evolution of neural rendering techniques beyond interactive applications. Future research could explore integrating more advanced view-dependent models to improve reflectivity and transparency for even richer visual outputs.

As rendering requirements continue to expand due to increasing demand for realistic virtual content, MERF's contributions position it as a fundamental framework for achieving efficiency without compromising the quality of visual outputs.