SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration (2312.07541v3)

Abstract: Recent techniques for real-time view synthesis have rapidly advanced in fidelity and speed, and modern methods are capable of rendering near-photorealistic scenes at interactive frame rates. At the same time, a tension has arisen between explicit scene representations amenable to rasterization and neural fields built on ray marching, with state-of-the-art instances of the latter surpassing the former in quality while being prohibitively expensive for real-time applications. In this work, we introduce SMERF, a view synthesis approach that achieves state-of-the-art accuracy among real-time methods on large scenes with footprints up to 300 m$^2$ at a volumetric resolution of 3.5 mm$^3$. Our method is built upon two primary contributions: a hierarchical model partitioning scheme, which increases model capacity while constraining compute and memory consumption, and a distillation training strategy that simultaneously yields high fidelity and internal consistency. Our approach enables full six degrees of freedom (6DOF) navigation within a web browser and renders in real-time on commodity smartphones and laptops. Extensive experiments show that our method exceeds the current state-of-the-art in real-time novel view synthesis by 0.78 dB on standard benchmarks and 1.78 dB on large scenes, renders frames three orders of magnitude faster than state-of-the-art radiance field models, and achieves real-time performance across a wide variety of commodity devices, including smartphones. We encourage readers to explore these models interactively at our project website: https://smerf-3d.github.io.

• The paper introduces SMERF, a novel method that splits large scenes into hierarchical submodels to enable memory-efficient, real-time rendering.
• It employs a distillation training strategy where a high-fidelity teacher NeRF guides a student model, ensuring high-quality rendering and smooth transitions.
• Experimental results demonstrate that SMERF achieves or surpasses current real-time methods, making photorealistic scene exploration accessible on everyday devices.

Introduction

Recent technological advancements have seen a significant leap in real-time view synthesis quality and speed. One of the fundamental challenges in this domain is reconciling highly detailed scene representations with the demands of interactive frame rates, especially for large, complex scenes. While explicit representations like meshes and point clouds have traditionally been used for this purpose, neural fields, particularly Neural Radiance Fields (NeRFs), have shown remarkable results in rendering photorealistic scenes—albeit at the cost of high computational resources, making them less feasible for real-time applications.

SMERF: A Scalable Approach to Radiance Fields

Addressing the need for real-time rendering capabilities, the newly developed method titled SMERF (Streamable Memory Efficient Radiance Fields) provides a scalable solution to view synthesis of large-scale scenes. SMERF utilizes a novel hierarchical model architecture consisting of multiple submodels that increase rendering capacity while constraining resource usage. Each submodel is specialized for a specific region of the scene, thus requiring only a fraction of the submodels to be active during rendering.

Additionally, SMERF applies a distillation training strategy whereby the model learns from a "teacher" NeRF that has already mastered rendering the scene at high fidelity, but at a slower pace not suitable for real-time applications. This allows the "student" SMERF model to adopt the same high-quality rendering capabilities and maintain coherence throughout the scene when transitioning from one submodel to another.

Real-time Rendering Across Devices

SMERF's ingenuity lies in its ability to run seamlessly on a wide variety of devices, including those with limited resources like smartphones and laptops. Experimental results indicate that SMERF not only meets but in some cases surpasses the current best real-time methods in terms of view synthesis fidelity—closing in on the quality of offline methods that are considered state-of-the-art. Remarkably, the method delivers these results while keeping rendering times in a real-time frame and adhering to memory constraints imposed by everyday consumer electronics.

Conclusion

The innovation presented by SMERF ushers in a new era of possibility for real-time exploration of large-scale 3D scenes. With its memory-efficient rendering that doesn't sacrifice image quality or rendering speed, SMERF stands as a significant step forward in the field of interactive 3D graphics and virtual exploration. Whether for gaming, virtual tours, or other interactive applications, SMERF offers a powerful tool to render detailed and immersive 3D environments in real time on standard consumer hardware.