UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (2312.13285v2)

Abstract: Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: \url{https://fangjinhuawang.github.io/UniSDF}.

• The paper introduces UniSDF, which unifies camera view and reflection radiance fields to accurately reconstruct complex scenes with reflective surfaces.
• It employs a progressive multi-resolution backbone combined with hash grids to accelerate training while enhancing geometric and surface fidelity.
• Experiments on datasets like DTU and Shiny Blender show UniSDF outperforms state-of-the-art methods in key metrics such as PSNR and SSIM.

Background and Challenges in 3D Reconstruction

Current techniques for neural 3D scene reconstructions have shown promise in creating representations from 2D images. However, they encounter difficulties when faced with complex scenes, especially those containing reflective surfaces. These challenges underscore the need for improved methods that effectively handle such intricate surfaces in large-scale scenes.

Introducing UniSDF: A Unified Approach

A novel approach titled "UniSDF" has been introduced, designed to more accurately reconstruct scenes with reflective properties. It combines two different radiance field parameterizations—one based on the camera view and another on reflections. The method includes a multi-resolution backbone trained progressively from coarse to fine resolution, leading to more refined and geometrically accurate surface reconstruction, especially for reflective objects.

Methodology and Innovation

The core innovation lies in combing the camera view-based radiance field and the reflection-based radiance field into a unified framework. Extensive experimentation has shown that this combination robustly reconstructs intricate scenes with varying degrees of reflectiveness. A new algorithm employs hash grids to speed up training while ensuring high-quality reconstruction.

Performance and Findings

The UniSDF has surpassed existing methods on several datasets, including DTU, Shiny Blender, Mip-NeRF 360, and the Ref-NeRF real dataset, demonstrating superior ability to reconstruct complex scenes with detailed, reflective surfaces. The approach showcases a blend of two radiance fields, favoring the reflection radiance field for modeling strong specularities and the camera view radiance field for more diffuse reflections.

The technique's strength is supported by quantitative measures like peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as qualitative comparisons that illustrate the improved reconstruction fidelity. Through an adaptable combination of radiance fields and a hash grid backbone, UniSDF has established a new state-of-the-art for reconstructing and rendering challenging scenes with precision.