NoPose-NeuS: Jointly Optimizing Camera Poses with Neural Implicit Surfaces for Multi-view Reconstruction (2312.15238v1)

Abstract: Learning neural implicit surfaces from volume rendering has become popular for multi-view reconstruction. Neural surface reconstruction approaches can recover complex 3D geometry that are difficult for classical Multi-view Stereo (MVS) approaches, such as non-Lambertian surfaces and thin structures. However, one key assumption for these methods is knowing accurate camera parameters for the input multi-view images, which are not always available. In this paper, we present NoPose-NeuS, a neural implicit surface reconstruction method that extends NeuS to jointly optimize camera poses with the geometry and color networks. We encode the camera poses as a multi-layer perceptron (MLP) and introduce two additional losses, which are multi-view feature consistency and rendered depth losses, to constrain the learned geometry for better estimated camera poses and scene surfaces. Extensive experiments on the DTU dataset show that the proposed method can estimate relatively accurate camera poses, while maintaining a high surface reconstruction quality with 0.89 mean Chamfer distance.

• The paper presents a joint optimization framework that refines camera poses and neural implicit surfaces for robust 3D reconstruction from multi-view images.
• It leverages a multi-layer perceptron encoding along with multi-view feature consistency and depth losses to enhance both pose accuracy and surface fidelity.
• Evaluations on the DTU dataset confirm that the method accurately reconstructs complex scenes without relying on pre-determined camera parameters.

Introduction to Neural Implicit Surfaces

The task of reconstructing 3D geometry from multi-view images presents significant challenges, particularly when dealing with complex structures such as non-Lambertian surfaces and thin objects. Classical Multi-view Stereo (MVS) methods have limitations in handling areas lacking clear texture, while recent neural rendering approaches impose stringent requirements on camera parameter knowledge.

Camera Pose Challenges and Innovations

In 3D geometry reconstruction, a typical condition is to have precise camera poses, which is often unachievable with casually captured images. Some innovative methods have addressed this by optimizing camera parameters alongside rendering. The paper under consideration extends this idea to Neural Implicit Surface (NeuS) models by proposing a method for simultaneous camera pose and scene geometry optimization. This approach introduces additional loss functions that cater to reinforcing the accuracy of the estimated camera positions and the fidelity of the scene's surfaces.

Methodology and Contributions

The key aspects of the proposed method involve encoding camera poses into a multi-layer perceptron (MLP), then refining these poses through multi-view feature consistency and depth losses. Qualitatively and quantitatively evaluated on the DTU dataset, the method showcases impressive accuracy in both camera pose estimation and surface reconstruction, displaying an ability to maintain high reconstruction quality while eschewing reliance on pre-determined camera parameters.

Findings and Further Potential

Demonstrated results offer a comprehensive analysis, confirming the method's proficiency in estimations and reconstructions. Although the camera pose initializations bear significant weight on the outcome, future research angles could involve refining this dependency or optimizing intrinsic camera parameters for a more complete calibration. The path paved by this method opens possibilities for diverse applications, particularly in scenarios where access to precise camera parameters is impractical.