DeDoDe: Detect, Don't Describe -- Describe, Don't Detect for Local Feature Matching (2308.08479v3)

Abstract: Keypoint detection is a pivotal step in 3D reconstruction, whereby sets of (up to) K points are detected in each view of a scene. Crucially, the detected points need to be consistent between views, i.e., correspond to the same 3D point in the scene. One of the main challenges with keypoint detection is the formulation of the learning objective. Previous learning-based methods typically jointly learn descriptors with keypoints, and treat the keypoint detection as a binary classification task on mutual nearest neighbours. However, basing keypoint detection on descriptor nearest neighbours is a proxy task, which is not guaranteed to produce 3D-consistent keypoints. Furthermore, this ties the keypoints to a specific descriptor, complicating downstream usage. In this work, we instead learn keypoints directly from 3D consistency. To this end, we train the detector to detect tracks from large-scale SfM. As these points are often overly sparse, we derive a semi-supervised two-view detection objective to expand this set to a desired number of detections. To train a descriptor, we maximize the mutual nearest neighbour objective over the keypoints with a separate network. Results show that our approach, DeDoDe, achieves significant gains on multiple geometry benchmarks. Code is provided at https://github.com/Parskatt/DeDoDe

• The paper introduces the DeDoDe framework that decouples keypoint detection from descriptor learning to enhance 3D consistency.
• It employs a semi-supervised two-view detection strategy and a mutual nearest neighbor objective for robust descriptor training.
• Experiments on MegaDepth-1500 and the Image Matching Challenge 2022 demonstrate significant gains in AUC and mAA metrics.

Overview of DeDoDe: Keypoint Detection for 3D Reconstruction

This essay examines the paper "DeDoDe: Detect, Don't Describe Describe, Don't Detect for Local Feature Matching" by Johan Edstedt et al., which introduces a new approach to keypoint detection critical for 3D reconstruction tasks. The authors tackle the challenge of detecting 3D-consistent keypoints across different views, independent of descriptor-based proxy tasks, which may not guarantee true 3D consistency.

Methodology

The core contribution of this work is the DeDoDe framework, which decouples the detection and description phases within the keypoint detection process for Structure from Motion (SfM). The authors identify the limitations of traditional joint learning approaches that bind keypoints to specific descriptors and instead propose a two-pronged strategy:

1. Keypoint Detection (Detect, Don't Describe): DeDoDe learns detectors using ground-truth 3D tracks from SfM reconstructions, avoiding reliance on the usual descriptor-driven nearest neighbors proxy. The approach incorporates a semi-supervised two-view detection objective that expands from a sparse set of keypoints to a more comprehensive detection set. This framework enhances precision and recall beyond initial detectors by directly optimizing 3D consistency.
2. Descriptor Learning (Describe, Don't Detect): Separately, descriptors are trained using a mutual nearest neighbor objective. Two descriptor models, DeDoDe-B and DeDoDe-G, are introduced, with the latter incorporating DINOv2 features to accommodate complex structures requiring more extensive contextual understanding.

Results and Performance

The DeDoDe method introduces substantial improvements in various geometry benchmarks. Notably, it bridges the performance gap between traditional detector-descriptor models and contemporary end-to-end matching frameworks. The extensive evaluations demonstrate that DeDoDe achieves strong performance metrics, with significant gains in benchmarks like MegaDepth-1500 and the Image Matching Challenge 2022.

• MegaDepth-1500: DeDoDe exhibits a notable improvement with an AUC$@5^{\circ}$ of 52.8 compared to 41.9 from prior state-of-the-art models such as ALIKED.
• Image Matching Challenge 2022: The approach showcases its robustness and generalization capabilities, achieving competitive mAA$@10$ scores alongside advanced graph neural network-based matchers.

Implications and Future Directions

The decoupled training of detectors and descriptors as proposed in DeDoDe suggests a shift in how feature matching pipelines might be constructed, focusing on modularity and adaptability. This work supports the notion that detectors can be trained independently to improve generalizability and robustness across different tasks.

The practical implications of DeDoDe are significant for 3D computer vision applications such as augmented reality, robotics, and autonomous navigation, where robust and efficient keypoint detection is essential.

Future research could explore:

• Integrating advanced augmentation techniques to improve DeDoDe's generalization in diverse settings.
• Enhancing the detector's scalability and efficiency to handle more challenging viewpoints and illumination changes.
• Experimenting with alternative datatypes (e.g., LiDAR) for keypoint detection.

This work presents a coherent argument for revisiting and revising the traditional interdependent learning strategies for detectors and descriptors, thereby paving the way for more versatile systems in 3D reconstruction contexts.