Sparse Deformable Descriptor Head (SDDH)

• The paper introduces SDDH, a keypoint-specific deformable descriptor head that learns adaptive sampling offsets to extract efficient, geometrically invariant descriptors.
• It uses a lightweight two-layer network and differentiable keypoint detection to combine deformable convolutions with sparse feature extraction, reducing GPU memory demands by 3x.
• Empirical results show that SDDH improves matching accuracy and reconstruction performance across tasks like homography, relocalization, and scale variations.

The Sparse Deformable Descriptor Head (SDDH) is a neural architecture component designed to efficiently extract expressive, geometrically invariant descriptors only at sparse keypoint locations in visual data. Introduced within the ALIKED network, SDDH addresses limitations of conventional convolutional operations in producing descriptors robust to geometric variation by leveraging a deformable sampling and aggregation mechanism specific to keypoints. This design eschews dense descriptor maps in favor of a lightweight, flexible, and memory-efficient sparse construction that enables state-of-the-art performance on tasks such as image matching, 3D reconstruction, and visual relocalization (Zhao et al., 2023).

1. Architectural Integration in ALIKED

SDDH functions as the descriptor extraction head within the ALIKED pipeline, which is structured in three primary stages:

1. Feature Encoding: Four sequential blocks produce multi-scale feature maps $$\{\mathbf F_1,\mathbf F_2,\mathbf F_3,\mathbf F_4\}$$, with deeper blocks employing $3 \times 3$ deformable convolutions (DCN v2) under SELU activations.
2. Feature Aggregation: Upsampling and $1 \times 1$ projections yield common-resolution feature maps $\mathbf F_i^u$, concatenated to form $\mathbf F \in \mathbb R^{H \times W \times C}$.
3. Keypoint and Descriptor Extraction:
   • The Score Map Head (SMH) predicts a dense score map $\mathbf S$.
   • Differentiable Keypoint Detection (DKD) applies non-maximum suppression and soft-argmax to select $N$ subpixel keypoints $\{\mathbf p_k\}$.
   • SDDH operates only at these $N$ locations: for each $\mathbf p_k$, it generates a descriptor $\mathbf d_k \in \mathbb R^{\mathrm{dim}}$ without constructing a full $H \times W \times \mathrm{dim}$ dense tensor as in traditional Descriptor Map Heads (DMH).

This design choice eliminates the memory and runtime burden associated with dense descriptor map computation, realizing a $3\times$ or greater reduction in GPU memory requirements and significant speed-ups compared to dense approaches.

2. Deformable Position Learning at Sparse Keypoints

At the core of SDDH is the learning of deformable support positions for each detected keypoint. Standard deformable convolution at location $\mathbf x$ is expressed as:

$$\mathbf F'(\mathbf x) = \sum_{i=1}^{K^2} w(\mathbf p_i)\, \mathbf F(\mathbf x + \mathbf p_i + \Delta\mathbf p_i)$$

where $\{\mathbf p_i\}$ define a regular $K \times K$ sampling grid, $w(\mathbf p_i)$ are learnable weights, and $\Delta\mathbf p_i$ are learned offsets. SDDH generalizes this by permitting $M$ freely located sample positions per keypoint, rather than a fixed grid.

For a keypoint $\mathbf p_k$:

• The $K \times K$ feature patch $\mathbf F_k$ centered at $\mathbf p_k$ is extracted.
• A lightweight two-layer network predicts $M$ sampling offsets $\{\Delta\mathbf p_{k,i}\}_{i=1}^M$:

$$\begin{aligned} \mathbf z_k &= \mathrm{SELU}\left(\mathrm{Conv}_{K\times K}(\mathbf F_k)\right) \ \mathbf o_k &= \mathrm{Conv}_{1\times1}(\mathbf z_k) \in \mathbb R^{2M} \end{aligned}$$

• Bilinear interpolation is used to sample features at $\mathbf p_k + \Delta\mathbf p_{k,i}$.

These sampled locations permit the descriptor to adaptively gather context supporting geometric invariance, as the positions are learned to maximize downstream matching performance.

3. Descriptor Synthesis and Aggregation

The sampled features for each keypoint are passed through a lightweight MLP (implemented as $1\times 1$ convolution plus SELU):

$$\Phi(\mathbf x) = \mathrm{SELU}(\mathrm{Conv}_{1\times1}(\mathbf x)) \in \mathbb R^{C'}$$

The final keypoint descriptor is computed as a weighted sum:

$$\mathbf d_k = \sum_{i=1}^M w_{k,i}\, \Phi(\mathbf F(\mathbf p_k+\Delta\mathbf p_{k,i}))$$

The weights $w_{k,i}$, obtained via a “convM” $1\times1$ aggregation, sum to 1. Optionally, descriptors are $L_2$-normalized. This aggregation scheme enables each descriptor vector to leverage maximally informative local feature content, with support positions dynamically adapted per keypoint.

4. Sparse Neural Reprojection Error Loss

SDDH descriptors are supervised using a sparse variant of the neural reprojection error (sNRE):

• For a matching image pair $(A, B)$ with keypoints $\{\mathbf p^A_i\}, \{\mathbf p^B_j\}$ and descriptors $\{\mathbf d^A_i\}, \{\mathbf d^B_j\}$, ground-truth matches are found via camera geometry.
• The indicator $P_{ij} = q_r(\mathbf p^A_i,\mathbf p^B_j)$ marks ground-truth matches.
• A matching similarity matrix $$[S^A]_{ij} = \langle\mathbf d^A_i,\mathbf d^B_j\rangle$$ is converted to a softmax distribution $Q_{ij}$.
• sNRE loss is then defined as the cross-entropy between ground-truth and predicted match distributions:

$$\mathcal L_{\mathrm{sNRE}} = -\sum_{i}\log\, Q_{i,\,j^*(i)}$$

where $j^*(i)$ is the ground-truth correspondence for $i$. This sparse supervision operates only over detected keypoints, reducing computational complexity compared to dense losses. The overall loss is a weighted sum of sNRE, keypoint reprojection loss, dispersity-peak loss, and reliability loss, with hyperparameters $\omega_{rp}=1$, $\omega_{pk}=0.5$, $\omega_{ds}=5$, $\omega_{re}=1$.

5. Training Data and Optimization Protocol

ALIKED and SDDH are trained on diverse datasets:

• MegaDepth (COLMAP-reconstructed images) for general perspective variations.
• R2D2-style Oxford, Paris, and Aachen pairs for homographic and stylized transformations.

Preprocessing includes resizing images to $800 \times 800$ pixels, with batch size $2$ (using gradient accumulation), and a total of $100{,}000$ training steps. Each image employs a keypoint budget of $400+400$, selected via non-maximum suppression. Training uses Adam optimization with $\beta_1 = 0.9$, $\beta_2 = 0.999$.

6. Empirical Performance and Ablation Studies

SDDH achieves strong quantitative results across multiple tasks, with ALIKED offered in three model sizes (Tiny, Normal, N(32)). Table 1 summarizes key metrics:

Model	Parameters (M)	Homography MMA@3 (%)	Homography MHA@3 (%)	FPS	Stereo mAA(10°) (%)	Relocalization 0.5 m/5° (%)
ALIKED-T(16)	0.192	72.99	78.70	125.9	–	–
ALIKED-N(16)	0.677	74.43	77.22	77.4	52.28	–
ALIKED-N(32)	–	–	–	–	–	88.8

Replacing the regular sparse head (SDH3) with SDDH$_{K=3,M=16}$ yields increases in Hpatches MS@3 (45.50% → 46.62%), MHA@3 (75.19% → 76.85%), and IMW mAA(10°) (63.58% → 65.39%), at a cost of $0.45$ GFLOPs extra. Increasing $M$ from 16 to 32 results in further increments (MS@3 → 47.37%, mAA(10°) → 67.78%).

Rotation augmentation leads to $>80\%$ matching accuracy up to $\pm30^\circ$. On scale variation, the single-scale ALIKED-N(16) outperforms all baselines up to $4\times$ scale changes.

7. Significance and Methodological Distinction

SDDH represents a methodological advance in keypoint-centric, deformable descriptor construction, balancing geometric flexibility and computational efficiency. By localizing deformable aggregation solely at detected keypoints and employing a novel sparse NRE loss, SDDH achieves strong matching and reconstruction performance with minimal memory footprint and high inference speed.

A plausible implication is that sparse, keypoint-specific deformable support substantially closes the accuracy gap with much heavier dense-map models, rendering compact descriptors feasible for real-time and resource-constrained applications. Moreover, the per-keypoint offset-prediction offers an avenue for future research into context-aware and adaptively sampled local representations in 2D and 3D vision systems (Zhao et al., 2023).