Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method (2312.12726v1)

Abstract: Neural radiance field (NeRF) enables the synthesis of cutting-edge realistic novel view images of a 3D scene. It includes density and color fields to model the shape and radiance of a scene, respectively. Supervised by the photometric loss in an end-to-end training manner, NeRF inherently suffers from the shape-radiance ambiguity problem, i.e., it can perfectly fit training views but does not guarantee decoupling the two fields correctly. To deal with this issue, existing works have incorporated prior knowledge to provide an independent supervision signal for the density field, including total variation loss, sparsity loss, distortion loss, etc. These losses are based on general assumptions about the density field, e.g., it should be smooth, sparse, or compact, which are not adaptive to a specific scene. In this paper, we propose a more adaptive method to reduce the shape-radiance ambiguity. The key is a rendering method that is only based on the density field. Specifically, we first estimate the color field based on the density field and posed images in a closed form. Then NeRF's rendering process can proceed. We address the problems in estimating the color field, including occlusion and non-uniformly distributed views. Afterward, it is applied to regularize NeRF's density field. As our regularization is guided by photometric loss, it is more adaptive compared to existing ones. Experimental results show that our method improves the density field of NeRF both qualitatively and quantitatively. Our code is available at https://github.com/qihangGH/Closed-form-color-field.

• The paper introduces an adaptive regularization technique that employs closed-form color field estimation to mitigate shape-radiance ambiguity in Neural Radiance Fields.
• It leverages spherical harmonics, transmittance weighting, and a residual estimation scheme to handle occlusions and non-uniform view distributions.
• Experimental results on DTU, NeRF Synthetic, and LLFF datasets show improved scene geometry with higher PSNR and reduced artifacts.

Overview

The paper introduces an adaptive regularization method targeted at addressing the ambiguity between shape and radiance in Neural Radiance Fields (NeRF), which is a prominent technique in computer graphics and vision for synthesizing realistic images from a 3D scene. This ambiguity often results in inaccuracies in the geometry of synthesized scenes leading to subpar novel views, particularly when only the photometric loss guides the learning process. The new method focuses on estimating a color field exclusively based on the density field and incorporates this into regularizing the training of NeRF.

Color Field Estimation

The researchers present a closed-form method to estimate the color field given the density field and a collection of posed images of a scene. This approach also tackles challenges including handling occlusions and factoring in non-uniform distributions of sample views. The method operates by first approximating the color at a point in a scene as a combination of spherical harmonics (SH) basis functions, and then it estimates SH coefficients. Adjustments are made to this estimation process to mitigate biases due to occlusion and non-uniform view distributions through transmittance-weighting and a residual estimation scheme, respectively.

Regularization Strategy

Using the estimated color field, the method introduces a new regularization term called the Closed-form Photometric (CF) loss. This term complements the conventional photometric loss during training, thus providing a more specific and adaptive form of supervision to the density field. In effect, it allows for the correction of geometric errors without being hampered by the limitations inherent to the NeRF's shape-radiance coupling.

Experimental Validation

Experiments conducted with the proposed method show that it not only results in qualitative improvements – such as sharper scene geometry and removal of artifacts – but it also quantitatively enhances the performance of explicit NeRF models. These improvements are demonstrated on the DTU, NeRF Synthetic, and LLFF datasets and achievements are measured using metrics such as PSNR and a novel metric called Inverse Mean Residual Color (IMRC).

Conclusion and Future Work

The new method proves to be an effective solution to the shape-radiance ambiguity in NeRF, managing to correct geometric errors and enhance synthesized novel views. However, the method has limitations when dealing with highly reflective objects that demand higher SH degrees to recover more details. Future work could further address these limitations and possibly remove the dependency on a parameterized color field altogether, focusing only on training and storing the density field, which could improve storage and computational efficiency.