DaD Keypoint Detector

• The paper introduces a reinforcement learning framework that unifies distinct light and dark detector modes through point-wise maximum knowledge distillation.
• It uses a VGG11-based encoder with a stride-aware, depthwise-separable decoder and a balanced top-K sampling strategy to ensure diverse and repeatable keypoints.
• Quantitative results demonstrate that DaD outperforms methods like SIFT and SuperPoint on SfM, pose estimation, and homography benchmarks.

DaD (Distilled Reinforcement Learning for Diverse Keypoint Detection) is a self-supervised, descriptor-free keypoint detector designed to produce diverse and highly repeatable interest points in images. It employs a reinforcement learning (RL) formulation that bypasses the need for descriptors, leveraging a policy-gradient approach and a balanced top-K sampling strategy. During training, it discovers two distinct detector modes—“light” (selecting high-intensity pixels) and “dark” (selecting low-intensity pixels)—and unifies them through point-wise maximum knowledge distillation, achieving state-of-the-art performance across several Structure-from-Motion (SfM), pose estimation, and homography benchmarks (Edstedt et al., 10 Mar 2025).

1. Model Architecture

DaD processes input images $I \in \mathbb{R}^{H \times W \times 3}$ using a backbone-encoder and stride-aware decoder architecture:

• Encoder: Based on VGG11, truncated at strides $\{1, 2, 4, 8\}$, producing feature maps with channels $\{64, 128, 256, 512\}$.
• Decoder: For each stride $s \in \{1,2,4,8\}$, three blocks are applied: DepthwiseConv $5\times5$ $\rightarrow$ BatchNorm $\rightarrow$ ReLU $\rightarrow$ PointwiseConv $1\times1$.
  • Outputs:
  • 1. Context feature of $C_s$ channels (upsampled and fused at the next stride).
  • 2. Score-map $\{1, 2, 4, 8\}$0 (logits per location).
• Final Output: After upsampling, produces a single-channel score-map $\{1, 2, 4, 8\}$1, interpreted as logits for a discrete pixel distribution. The detector’s policy is $\{1, 2, 4, 8\}$2.

This architecture follows the DeDoDe-S model (VGG11 encoder and depthwise-separable decoder blocks as in DeDoDe v2).

2. Reinforcement Learning Formulation

DaD’s detection objective is fully self-supervised and RL-based. The RL setup is characterized by:

• State: An image pair $\{1, 2, 4, 8\}$3 and their pseudo-ground-truth depths $\{1, 2, 4, 8\}$4, along with the current detector $\{1, 2, 4, 8\}$5.
• Action Space: Top-K deterministic selection, $\{1, 2, 4, 8\}$6, drawn from $\{1, 2, 4, 8\}$7.
• Policy Network: Defined as $\{1, 2, 4, 8\}$8 via softmax on $\{1, 2, 4, 8\}$9.
• Reward: Two-view repeatability. Each matched pair $\{64, 128, 256, 512\}$0 is assessed as

$\{64, 128, 256, 512\}$1

where $\{64, 128, 256, 512\}$2 if $\{64, 128, 256, 512\}$3, $\{64, 128, 256, 512\}$4 otherwise, with threshold $\{64, 128, 256, 512\}$5 of image height. $\{64, 128, 256, 512\}$6 denotes geometric projection via the two-view depth.

To stabilize gradients:

$\{64, 128, 256, 512\}$7

The RL objective is optimized as minimization:

$\{64, 128, 256, 512\}$8

3. Balanced Top-K Sampling and Diversity Enforcement

Naively sampling from $\{64, 128, 256, 512\}$9 can lead to mode collapse, focusing detection on few dense regions. DaD introduces a deterministic, “balanced” top-K sampling procedure:

1. Policy Smoothing: $s \in \{1,2,4,8\}$0, with Gaussian $s \in \{1,2,4,8\}$1 ($s \in \{1,2,4,8\}$2 of the diagonal).
2. Balanced Distribution: $s \in \{1,2,4,8\}$3.
3. Non-Maximum Suppression: $s \in \{1,2,4,8\}$4.
4. Top-K Selection: The highest-scoring $s \in \{1,2,4,8\}$5 pixels from $s \in \{1,2,4,8\}$6 are selected as keypoints.

This enforces (a) spatial sparsity (via NMS), (b) diversity (down-weighting of dense clusters by KDE), and (c) selection of globally top-scoring keypoints.

4. Emergence of Light and Dark Detectors

Training with $s \in \{1,2,4,8\}$7, including rotation augmentation, leads to two distinct optima:

• Light Detectors: Select high-intensity (bright) pixels.
• Dark Detectors: Select low-intensity (dark) pixels.

Both achieve comparable two-view repeatability rewards, but each omits repeatable keypoints of the opposite intensity type. This indicates local optima in the RL objective correlated with intensity distributions.

5. Knowledge Distillation via Point-wise Maximum (“DaD”)

To capture both light and dark interest points within a single model, DaD employs a point-wise maximum knowledge distillation scheme:

• Let $s \in \{1,2,4,8\}$8 and $s \in \{1,2,4,8\}$9 be the trained light and dark detectors (held fixed).
• Define the “expert” distribution $5\times5$0.
• The DaD detector $5\times5$1 is trained to minimize the Kullback-Leibler divergence:

$5\times5$2

This produces a unified detector with coverage over both intensity regimes.

6. Training and Implementation Details

• Training Phases:

1. Train a dark detector: 600k pairs ($5\times5$3), update via AdamW.
2. Train a light detector: analogous, 800k pairs.
3. Distill DaD detector from both: 800k pairs.

• Implementation:
  • Dataset: MegaDepth, with random rotations ($5\times5$4), 640 px resolution.
  • Sampling budget: $5\times5$5 keypoints.
  • Reward/matching threshold: $5\times5$6 of image height.
  • Regularization: $5\times5$7 (where $5\times5$8 indicates co-visible regions and $5\times5$9 is a Gaussian with $\rightarrow$0 px).
  • Inference resizes the longer side to 1024 px, uses NMS $\rightarrow$1, and sub-pixel refinement by local softmax in $\rightarrow$2 with temperature $\rightarrow$3.

7. Quantitative Results and Comparative Evaluation

Ablation and benchmark results demonstrate strong performance:

Benchmark	Method	AUC@5° (Essential, K=512)	AUC@5° (Fundamental, K=512)
MegaDepth1500	SIFT	48.9	36.7
	SuperPoint	57.5	41.7
	ReinforcedFP	58.7	42.7
	ALIKED	63.8	49.7
	DeDoDe v2	57.5	40.8
	DaD	64.9	50.6
ScanNet1500	DeDoDe v2	19.9	10.3
	ALIKED	22.3	14.2
	DaD	25.7	18.3
HPatches (AUC@3 px, K=512, Homography)	DeDoDe v2	52.2	–
	ALIKED	52.6	–
	DaD	58.2	–

• Ablation (MegaDepth) AUC@5° pose:
  • Light only: 50.2 (512 kp) → 55.9 (8192 kp)
  • Dark only: 50.0 → 55.7
  • Distill (mean): 50.7 → 56.0
  • Distill $\rightarrow$4: 51.0 → 56.3
  • Distill $\rightarrow$5 (DaD): 51.0 → 56.5 (best)
• Runtime (A100, 512 kp): ALIKED: 8.9 ms; DISK: 14.3 ms; SuperPoint: 6.5 ms; DeDoDe v2: 42.8 ms; DaD: 18.7 ms.

8. Significance and Application Context

DaD establishes a new state-of-the-art in descriptor-free, fully self-supervised keypoint detection for SfM, pose estimation, and homography tasks, removing descriptor dependence and encouraging mode diversity in detector outputs. It demonstrates that reinforcement learning objectives, with appropriate diversity enforcement and knowledge distillation, can yield detectors highly effective for downstream geometric vision tasks (Edstedt et al., 10 Mar 2025). The discovery of mode specialization and the unification by maximal distillation suggest that complementary interest point types are essential for complete geometric scene understanding.