Cameras as Rays: Pose Estimation via Ray Diffusion

Abstract: Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views (<10). In contrast to existing approaches that pursue top-down prediction of global parametrizations of camera extrinsics, we propose a distributed representation of camera pose that treats a camera as a bundle of rays. This representation allows for a tight coupling with spatial image features improving pose precision. We observe that this representation is naturally suited for set-level transformers and develop a regression-based approach that maps image patches to corresponding rays. To capture the inherent uncertainties in sparse-view pose inference, we adapt this approach to learn a denoising diffusion model which allows us to sample plausible modes while improving performance. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D while generalizing to unseen object categories and in-the-wild captures.

• The paper introduces a novel ray-based representation that redefines camera pose estimation using denoising diffusion models.
• It employs a transformer-based architecture to regress ray representations, converting them into precise camera parameters with enhanced generalization.
• Experimental results show improved rotation and center accuracies, even for in-the-wild captures and unseen object categories.

Cameras as Rays: Pose Estimation via Ray Diffusion

Introduction

The paper "Cameras as Rays: Pose Estimation via Ray Diffusion" addresses the critical task of estimating camera poses from sparsely sampled views, a challenge in applications requiring high-fidelity 3D reconstructions. Previous approaches primarily rely on top-down prediction methods, focusing on global parameterizations of camera extrinsics, but the authors propose a novel distributed representation that treats a camera as a bundle of rays. This approach enhances pose precision by tightly coupling the representation with spatial image features and adapting denoising diffusion models to sample plausible modes while handling uncertainties inherent in sparse-view pose inference.

Methodology

Ray Representation

The authors redefine camera parametrization by treating cameras as collections of rays instead of predicting global rotations and translations. Plücker coordinates are employed to parameterize rays, providing a robust method for capturing camera parameters in distributed form.

Figure 1: Recovering Sparse-view Camera Parameters by Denoising Rays.

Conversion Between Representations

Converting traditional camera parameterizations into ray bundles involves unprojecting rays from camera centers to pixel coordinates. Conversely, the system solves optimization problems to revert ray bundles back into conventional camera parameters, enhancing the flexibility and precision of camera pose predictions.

Figure 2: Converting Between Camera and Ray Representations.

Regression and Diffusion Approach

A transformer-based architecture is used to predict ray representations, processing image patches to estimate corresponding rays. This regression-based method initially showed state-of-the-art improvements. The approach was further extended using denoising diffusion models to account for ambiguity and uncertainty in sparse visual data, optimizing L2 loss to refine ray predictions.

Figure 3: Denoising Ray Diffuser Network.

Experimental Results

The paper reports superior camera rotation and center accuracies against traditional methods like COLMAP and learning-based approaches such as RelPose and PoseDiffusion. The ray-based approaches efficiently generalize across both seen and unseen categories and demonstrate scalability when tested with larger image samples.

Figure 4: Qualitative Comparison Between Predicted Camera Poses.

Figure 5: Generalization to In-the-wild Self-captures.

Discussion

The proposed ray-based representation effectively handles the complexities of sparse-view camera pose estimation, offering significant improvements over existing methods. The ability to leverage this encoding for probabilistic modeling allows capturing multiple plausible distributions of camera poses, addressing inherent uncertainties due to symmetries and limited observations.

Figure 6: Modeling Uncertainty Via Sampling Modes.

The work presents substantial potential for future exploration in single-view and dense multi-view setups, with possibilities for integrating geometric consistency joint inference methods.

Conclusion

"Cameras as Rays: Pose Estimation via Ray Diffusion" introduces a promising framework for camera pose estimation in scenarios with sparse data availability, showcasing robust performance improvements and offering insights into the future of distributed camera representations. The approach redefines traditional parameterizations, enabling high precision and adaptability in 3D reconstructions and expanding the applications scope across unseen object categories and real-world scenarios.