Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS (2408.08723v1)

Abstract: Novel View Synthesis (NVS) without Structure-from-Motion (SfM) pre-processed camera poses--referred to as SfM-free methods--is crucial for promoting rapid response capabilities and enhancing robustness against variable operating conditions. Recent SfM-free methods have integrated pose optimization, designing end-to-end frameworks for joint camera pose estimation and NVS. However, most existing works rely on per-pixel image loss functions, such as L2 loss. In SfM-free methods, inaccurate initial poses lead to misalignment issue, which, under the constraints of per-pixel image loss functions, results in excessive gradients, causing unstable optimization and poor convergence for NVS. In this study, we propose a correspondence-guided SfM-free 3D Gaussian splatting for NVS. We use correspondences between the target and the rendered result to achieve better pixel alignment, facilitating the optimization of relative poses between frames. We then apply the learned poses to optimize the entire scene. Each 2D screen-space pixel is associated with its corresponding 3D Gaussians through approximated surface rendering to facilitate gradient back propagation. Experimental results underline the superior performance and time efficiency of the proposed approach compared to the state-of-the-art baselines.

• The paper introduces a correspondence-guided approach that integrates pose estimation with 3D Gaussian splatting, bypassing conventional SfM pre-processing.
• It leverages monocular depth for initialization and a learnable SE-3 transformation to optimize camera poses via a robust 2D-3D correspondence loss.
• Experimental results demonstrate enhanced PSNR, SSIM, and efficiency, confirming its potential for real-time novel view synthesis applications.

Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS

The paper "Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS" presents an innovative approach to novel view synthesis (NVS) that bypasses the usual structure-from-motion (SfM) pre-processed camera poses. This method addresses significant challenges by integrating pose optimization within an end-to-end framework, thereby mitigating the reliance on computationally expensive and error-prone SfM techniques.

Overview

NVS aims to generate new images of a scene from arbitrary viewpoints, which has applications ranging from virtual reality to autonomous systems. Conventional methods often depend on accurate camera poses obtained through SfM methods like COLMAP, which are not only time-consuming but also sensitive to feature extraction errors, especially in textureless or repetitive regions. This paper critiques existing SfM-free methodologies for their dependence on per-pixel image loss functions, which are susceptible to significant gradients and unstable optimization due to misalignment from inaccurate initial poses.

Methodology

The core contribution of this work is the Correspondence-Guided SfM-free 3D Gaussian Splatting (CG-3DGS) method. This approach utilizes correspondences detected between the target image and the rendered results to achieve better pixel alignment, thereby enhancing the optimization of relative poses between frames. The CG-3DGS approach can be divided into three main components:

1. Initialization from Monocular Depth: The method starts by generating a 3D Gaussian initialization from the monocular depth of the initial frame, avoiding the need for SfM-derived points.
2. Pose Estimation via 3D Gaussians Transformation: The relative camera pose is iteratively estimated through a learnable SE-3 affine transformation applied to the 3D Gaussians. This transformation is optimized by minimizing the correspondence-based loss between the rendered and target images.
3. Correspondence-based Loss Function: The proposed method uses off-the-shelf detectors to establish 2D correspondences, linking 2D screen-space disturbances to 3D surface points in a differentiable manner. This correspondence-based loss comprises color losses and depth alignment terms, enhancing both pixel-wise supervision and long-range motion robustness.

The paper introduces an approximated surface rendering technique to facilitate gradient back-propagation from 2D correspondences to 3D Gaussians, effectively bypassing the need for explicit surface reconstruction, which would otherwise be computationally prohibitive.

Experimental Evaluation

The experimental results on the Tanks and Temples and CO3D-V2 datasets clearly demonstrate the superior performance of the proposed CG-3DGS method. Key findings include:

• Quality of Novel View Synthesis: The method consistently achieves higher PSNR, SSIM, and lower LPIPS values compared to baseline methods such as CF-3DGS, Nope-NeRF, BARF, and NeRFmm. For instance, on the Tanks and Temples dataset, CG-3DGS exhibits substantial improvements, with PSNR values surpassing baseline methods by up to 3.5 points in certain scenes.
• Camera Pose Estimation: The camera pose estimation metrics, including Relative Pose Error (RPE) and Absolute Trajectory Error (ATE), show that CG-3DGS reaches performance levels close to COLMAP, even outperforming in certain scenarios. This is particularly notable given that CG-3DGS does not require pre-processed SfM poses.
• Efficiency: The proposed method also demonstrates efficiency in computational time, making it more suitable for real-time applications compared to traditional SfM-reliant methods.

Implications and Future Work

The CG-3DGS method has broad implications for the field of computer vision, particularly in scenarios where rapid adaptation and robustness are critical, such as autonomous navigation and AR/VR applications. The avoidance of SfM pre-processing steps not only reduces computational overhead but also mitigates potential failures due to poorly textured or repetitive regions.

Theoretically, this work introduces a paradigm shift by demonstrating the feasibility of high-fidelity NVS without relying on costly and error-prone pre-computed camera poses. Practically, the system's demonstrated proficiency in handling complex camera movements and significant performance gains hints at its applicability in real-world scenarios where quick and reliable deployment is crucial.

Looking forward, future research could explore further improvements in correspondence detection techniques and more refined loss functions that may capture even subtler nuances in scene understanding. Additionally, extending the CG-3DGS framework to other forms of 3D representations and exploring its scalability across diverse and larger datasets would be worthwhile endeavors.

Overall, the CG-3DGS approach signifies a significant advancement in the field of SfM-free NVS, offering a robust, efficient, and theoretically sound methodology that sets a new standard for future research in this domain.