Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences

Abstract: Given two images, we can estimate the relative camera pose between them by establishing image-to-image correspondences. Usually, correspondences are 2D-to-2D and the pose we estimate is defined only up to scale. Some applications, aiming at instant augmented reality anywhere, require scale-metric pose estimates, and hence, they rely on external depth estimators to recover the scale. We present MicKey, a keypoint matching pipeline that is able to predict metric correspondences in 3D camera space. By learning to match 3D coordinates across images, we are able to infer the metric relative pose without depth measurements. Depth measurements are also not required for training, nor are scene reconstructions or image overlap information. MicKey is supervised only by pairs of images and their relative poses. MicKey achieves state-of-the-art performance on the Map-Free Relocalisation benchmark while requiring less supervision than competing approaches.

• The paper introduces MicKey, a neural network that predicts 3D metric keypoints from 2D images to enable scale-metric relative pose estimation.
• It employs a fully differentiable, probabilistic framework that integrates descriptor similarities and keypoint confidence for robust correspondence selection.
• The approach achieves state-of-the-art results on the Map-Free Relocalisation benchmark while requiring less supervision compared to traditional methods.

Matching 2D Images in 3D: A Novel Approach with MicKey

Introduction

The task of estimating the relative camera pose between two images has been a cornerstone problem in computer vision, with direct applications in navigation, 3D reconstruction, and augmented reality (AR). Traditionally, this problem has been approached by matching keypoints between images to establish correspondences and subsequently estimating the pose up to a scale. However, applications that require a metric understanding of the scene, such as AR, demand scale-metric pose estimates. Traditional methods often fall short in this regard, necessitating methods that can predict metric correspondences directly from images.

In our latest work, we introduce MicKey, a keypoint matching pipeline that breaks ground by predicting metric 3D keypoints in camera space directly from 2D images. By doing so, MicKey enables the inference of scale-metric relative poses without the necessity of depth measurements, a significant advancement over existing approaches.

Methodology

MicKey leverages a neural network to learn matching 3D coordinates across images, thus facilitating metric relative pose estimation without direct depth measurements or scene reconstructions. By adopting a fully differentiable pipeline, including the Kabsch pose solver, the training process requires only pairs of images and their ground truth relative poses for supervision.

Key to our approach is treating the output of the network probabilistically, allowing for an end-to-end training strategy that is robust to inaccuracies in keypoint detection and descriptor matching. This probabilistic nature also extends to correspondence selection, where we integrate both descriptor similarities and keypoint confidence scores to determine the likelihood of matches.

Novel Contributions

Our work brings several innovations to the field of metric relative pose estimation:

1. Introduction of MicKey: A neural network capable of accurately predicting 3D metric keypoints from single 2D images, which, when matched across images, enable the computation of scale-metric relative poses.
2. Probabilistic Correspondence Selection: Through a novel application of probability theory to keypoint matching, we efficiently handle uncertainties inherent in feature matching processes.
3. End-to-end Differentiable Training: By treating elements of the pose estimation process probabilistically, we achieve an end-to-end training regime that only requires relative pose supervision, eliminating the need for direct depth measurements or extensive scene reconstructions.

Results and Implications

MicKey exhibits state-of-the-art performance on the Map-Free Relocalisation benchmark, surpassing contemporaneous approaches in metric relative pose estimation. Crucially, it requires less supervision than competing methods, showcasing the effectiveness of its end-to-end learning strategy. Our results underscore the potential for applying MicKey to real-world scenarios where scale-metric pose estimation is crucial. Moreover, the ability of MicKey to infer 3D information from 2D images without explicit depth measurements opens avenues for future research in unsupervised and semi-supervised learning domains within 3D computer vision.

Future Directions

The success of MicKey hints at promising research trajectories. One potential area of exploration is the application of these techniques to semantic matching tasks, where understanding the 3D structure of scenes can provide additional context for interpreting complex environments. Furthermore, investigating the integration of MicKey with IMU data or other sensor readings could yield even more robust pose estimation systems, particularly in challenging scenarios with limited visual features.

In conclusion, MicKey represents a significant step forward in the quest for accurate metric relative pose estimation from 2D images. By innovatively leveraging 3D keypoint predictions and a probabilistic, end-to-end trainable pipeline, we demonstrate the feasibility of scale-metric pose estimation without direct depth measurements, paving the way for future advancements in 3D computer vision and AR applications.

Detailed citations and references related to the work herein can be found in the full paper, “Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences,” available on the project's webpage.