BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2302.14859v2)

Abstract: We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.

• The paper presents a novel approach to optimize hybrid neural volume-surface representations for meshing real-time 3D scenes.
• It employs a modified Marching Cubes algorithm with vertex-based spherical Gaussian integration for efficient appearance modeling.
• Experimental results show improved PSNR, SSIM, and LPIPS metrics while achieving high FPS on commodity hardware for browser-based applications.

BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis

The paper "BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis" presents an innovative method for creating high-quality meshes from neural signed distance functions (SDFs) tailored for real-time view synthesis in large, unbounded scenes. Proposed by Lior Yariv et al., this research builds upon existing neural radiance field approaches, specifically addressing the challenges associated with real-time rendering on commodity hardware while preserving high levels of accuracy and quality in the generated 3D models.

Scene Representation and Optimization

A pivotal contribution of this paper is the optimization strategy for a hybrid neural volume-surface representation, which is meticulously designed to possess well-behaved level sets corresponding to surfaces within the scene. This representation is realized through a neural SDF framework inspired by VolSDF but adapted for unbounded scenes using the effective contracting approach from mip-NeRF 360. The optimization process involves constructing a detailed SDF using a variant of mip-NeRF 360, which ensures the density is parameterized appropriately to robustly define surfaces while suppressing unwarranted density formations in unobserved regions via visibility-aware mechanisms.

Mesh Extraction and Appearance Baking

In transforming the SDF representation into usable 3D meshes, the paper introduces a meshing method premised on the Marching Cubes algorithm. Notably, a visibility and free-space culling technique is incorporated to prevent the inclusion of extraneous surfaces, followed by an iterative region-growing procedure to fill potential gaps. Once extracted, the mesh strategically exploits contracted space to ensure the triangle density is optimized for visual perception — allowing smaller triangles nearer to the origin, where content detail is higher.

For appearance modeling, the vertex-based integration of spherical Gaussians (SGs) for representing view-dependent effects marks a departure from expensive MLPs traditionally used in NeRF-related methodologies. Spherical Gaussians facilitate efficient computation and interpolation in rasterized meshes, effectively reducing computational overhead while maintaining nuanced appearance characteristics such as specularity and material reflectance.

Performance and Practical Implications

The empirical results demonstrate superior performance compared to state-of-the-art real-time rendering methods, notably in PSNR, SSIM, and LPIPS metrics across both indoor and outdoor unbounded scenes. The method’s significant reduction in energy consumption and increased frames per second (FPS) — all achieved within browser-based environments on commodity hardware — critically underscores its practicality for applications requiring rapid and realistic 3D renderings.

Furthermore, the produced meshes are versatile enough to support common graphics applications, such as appearance editing and physics simulations, due to their high fidelity and compatibility with traditional graphics pipelines. This positions BakedSDF not merely as an advancement in mesh generation but as a practical approach to utilizing neural representations in dynamic and interactive graphics contexts.

Future Potential

While BakedSDF successfully addresses multiple limitations of contemporary NeRF technologies, such as render speed and hardware adaptability, it opens several avenues for future exploration. Enhancing the representation for semi-transparent materials and fine-detailed geometries remains an area for potential refinement. Additionally, strategies for compression and efficient on-disk storage of large-scale meshes could further refine the scalability of the approach.

In summary, BakedSDF acts as a robust framework for transforming neural SDFs into actionable, efficient 3D models conducive to real-time graphics applications. Its contributions lie in both methodological innovations and practical enablements, paving the way for continued advancements in neural graphics rendering.