No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (2410.24207v1)

Abstract: We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from \textit{unposed} sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available at https://noposplat.github.io/.

• The paper demonstrates that 3D scene reconstruction can be achieved without camera pose data by leveraging a feed-forward network trained solely with photometric loss.
• It introduces a canonical space approach where the first view’s local coordinates anchor the scene, effectively aligning 3D Gaussian primitives from sparse images.
• Experimental results show NoPoSplat outperforms traditional methods in novel view synthesis and relative pose estimation under limited image overlap conditions.

An Expert Overview of "No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images"

The paper "No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images" introduces a novel method called NoPoSplat, which addresses a key challenge in generalizable 3D reconstruction: the reliance on accurate camera poses. This method facilitates reconstructing 3D scenes from sparse and unposed multi-view images, leveraging a feed-forward network that operates in real-time, and is noteworthy for its use of only photometric loss during training, bypassing the traditional requirement for detailed camera pose data.

Approach and Technical Details

NoPoSplat is centered around a canonical space approach, wherein the local camera coordinates of the first input view are adopted as a reference for the entire scene. This innovative tactic diminishes the complexities associated with transforming Gaussian primitives from local coordinate systems to a global system, a common hindrance in previous methods. By anchoring one view in this canonical space, the approach ensures the 3D Gaussians generated are effectively aligned across different inputs, even with minimal images.

To resolve scale ambiguities, NoPoSplat employs different intrinsic embedding techniques, ultimately finding that converting camera intrinsics into a token embedding and coupling it with image tokens considerably improves scene scale prediction capabilities. This adjustment sets the stage for using the estimated 3D Gaussians in tasks such as novel view synthesis and relative pose estimation.

Numerical Results and Comparisons

Rigorous experiments validate the supremacy of NoPoSplat over state-of-the-art methods. Noteworthy is its capability to surpass pose-required methods in scenarios with limited image overlap, a traditionally challenging situation for 3D reconstruction tasks. Furthermore, NoPoSplat is also remarkably effective in pose estimation without direct ground truth depth information or explicit matching losses, spotlighting its robustness compared to prevalent state-of-the-art techniques.

Implications and Future Directions

Practically, NoPoSplat's contribution lies in its ability to eliminate the dependency on pre-calculated camera poses, significantly enhancing the method's applicability in real-world scenarios where obtaining dense video input or accurate poses is cumbersome or impractical. Theoretically, this approach opens potential pathways in the scalability and generalizability of 3D reconstruction models using large-scale video datasets, as depth ground truth is not a prerequisite.

While successful, the approach is currently constrained to static scenes. Extending this work to dynamic scenarios, possibly integrating aspects of temporal coherence and motion dynamics, presents an intriguing direction for future advancements. Additionally, further exploration of incorporating more sophisticated intrinsic encoding strategies or adopting unsupervised learning paradigms might lead to even broader applicability across diverse scene types and data distributions.

Conclusion

In summary, the introduction of NoPoSplat adds a significant dimension to the domain of 3D reconstruction by demonstrating a feasible, practical method that circumvents the need for exhaustive camera pose data. This paper provides a foundation for future research aiming to refine AI-driven 3D reconstruction methods while also enhancing their practical deployment capabilities under variable operational constraints.