Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis (2312.02255v3)

Abstract: Recent neural rendering and reconstruction techniques, such as NeRFs or Gaussian Splatting, have shown remarkable novel view synthesis capabilities but require hundreds of images of the scene from diverse viewpoints to render high-quality novel views. With fewer images available, these methods start to fail since they can no longer correctly triangulate the underlying 3D geometry and converge to a non-optimal solution. These failures can manifest as floaters or blurry renderings in sparsely observed areas of the scene. In this paper, we propose Re-Nerfing, a simple and general add-on approach that leverages novel view synthesis itself to tackle this problem. Using an already trained NVS method, we render novel views between existing ones and augment the training data to optimize a second model. This introduces additional multi-view constraints and allows the second model to converge to a better solution. With Re-Nerfing we achieve significant improvements upon multiple pipelines based on NeRF and Gaussian-Splatting in sparse view settings of the mip-NeRF 360 and LLFF datasets. Notably, Re-Nerfing does not require prior knowledge or extra supervision signals, making it a flexible and practical add-on.

• The paper presents a multi-stage NeRF enhancement that integrates synthetic view generation with epipolar constraints, significantly improving 3D novel view fidelity.
• The method employs a dual training process where a second NeRF model is retrained using both real and pseudo-views to enforce geometric consistency.
• Extensive experiments show that Re-Nerfing enhances reconstruction accuracy in both dense and sparse data scenarios without requiring additional external models.

Enhancing 3D Scene Reconstruction with Re-Nerfing

Creating stunningly realistic three-dimensional scenes from a collection of images is a popular application of AI technology. Specifically, Neural Radiance Fields (NeRFs) have revolutionized this field by synthesizing new perspectives on a scene that weren't captured in the original dataset. Nonetheless, this technology is not without its challenges – when limited data is available, artifacts and inaccuracies tend to creep into the 3D representations. Addressing these limitations, a recent development presents an ingenious approach: Re-Nerfing.

Re-Nerfing is a novel, multi-stage technique designed to enhance the output of NeRF models, particularly when they are fed sparser datasets. It takes great advantage of NeRF's inherent ability to synthesize views, building upon the original model to enforce geometric consistency and improve the quality of novel views. At its core, Re-Nerfing initially sticks to the standard procedure of training a NeRF model with available views. Following that, it generates additional pseudo-views that simulate a stereo or trifocal camera setup and retrains a second NeRF model using both the original and artificially generated images. During this retraining, the system integrates additional geometric constraints, pushing the scene's representation towards greater fidelity.

The key to Re-Nerfing's success is its enforcement of epipolar geometry constraints from synthetic views. These constraints guide the estimation of depth and density during the second round of NeRF model training, resulting in more accurate and geometrically-consistent synthetic views. Extensive experiments demonstrate that harnessing these synthetic views to retrain the model (Re-Nerfing) leads to noticeable enhancements even when the input scenarios are already dense.

Interestingly, results from Re-Nerfing improve along two axes. First, when training data is dense, Re-Nerfing refines novel views, particularly those with lower visibility in the training dataset. For less dense training scenarios, the benefits are even more significant, suggesting that the technique efficaciously mitigates issues arising from insufficient data.

Re-Nerfing's methodology doesn't just stop at enhancing scene fidelity; it also offers a novel density loss derived from epipolar geometry. This aspect is portable and can potentially boost any stereo-setup used in training NeRF models. Furthermore, Re-Nerfing doesn't lean on extraneous data or models. It generates and utilizes synthetic views solely based on the images already available to the baseline NeRF model, preserving the advantages of the original technology.

In practical terms, the Re-Nerfing approach holds promise for those looking to create detailed 3D models from limited visual data. It could transform applications wherein resources to capture comprehensive datasets are scarce, such as archaeological documentation and virtual reality content creation.

In terms of limitations, the current iteration of Re-Nerfing hinges on the quality of the base NeRF model's renderings – if the first-stage model doesn't generate reasonable scene geometry, the benefits of Re-Nerfing diminish. Also, the method isn't as potent in extremely sparse scenes, though blending it with other strategies targeting these scenarios could be a research path worth exploring. Additionally, the simple patch matching for enforcing geometric constraints might encounter issues in featureless or repetitive regions. Bringing in advanced feature matching strategies could help strengthen the technique further.

In conclusion, Re-Nerfing epitomizes an astute exploitation of NeRF's synthesis capabilities, guiding the technology to new heights of precision and efficiency. By turning NeRF's weaknesses into strengths, Re-Nerfing paves the way for more robust and detailed 3D scene reconstructions.