---
title: 'HardMatch: Challenging Matching Benchmark'
url: https://www.emergentmind.com/topics/hardmatch
type: topic
---

# HardMatch: Challenging Matching Benchmark

HardMatch is a hand-annotated, deliberately difficult two-view matching benchmark introduced in "LoMa: Local Feature Matching Revisited" [2604.04931]. Designed to address benchmark saturation in local feature matching, HardMatch provides large headroom for improvement by presenting challenging, diverse, and semantically varied image pairs accompanied by precise ground-truth correspondences.

## 1. Dataset Construction

HardMatch was constructed from a large corpus of ≈ 100,000 images scraped from Wikimedia Commons under permissive licenses. Images were organized into 100 semantic categories representing a broad spectrum of content types, including famous buildings, natural landmarks, industrial sites, and drawings. For each category, 100 candidate image pairs were randomly sampled.

An initial filtering step applied RoMa v2 to score the maximum match "confidence" of each pair. Pairs with confidence in the interval [0.3, 0.9]—interpreted as "difficult but matchable"—were retained, excluding trivially easy or impossible pairs. Subsequently, these pairs underwent manual inspection to ensure matchability, repeating the selection process until 10 matchable pairs per category were obtained, resulting in a final dataset of 1,000 curated pairs.

The dataset was split, randomly partitioning 10 categories (100 pairs) for validation and 90 categories (900 pairs) for testing. Images were pre-processed by rescaling to 640×640 pixels. Approximately 20 pairs containing dynamic (non-rigid) content were flagged and excluded from standard fundamental matrix-based evaluation.

## 2. Annotation Protocol

For each selected image pair, between 8 and 28 corresponding 2D points—referred to as "salient matches"—were manually annotated by the authors. Annotators were instructed to label every obvious repetitively-textured or geometrically salient feature in both images.

Quality control procedures included recruiting 8 independent annotators, each verifying 20 randomly selected pairs. Annotators were provided with one image point and tasked with clicking the corresponding point in the other image. The distribution of pixel-wise annotation error demonstrated a median annotation noise below ≈ 1 px, guiding the choice of minimum evaluation thresholds at or above 1 px.

## 3. Dataset Characteristics

HardMatch consists of 1,000 image pairs spanning 100 semantic categories. Each category constitutes approximately 10 pairs, grouped into 9 "challenge groups":

- Aerial vs Ground
- Celestial (earth–star; constellations)
- Doppelgängers (similar structures)
- Drawings (sketch→photo)
- Illumination (day⇄night)
- Nature (organic scenes)
- Seasonal (spring⇄winter)
- Temporal (century-apart)
- Viewpoint (including up to 180° flips)

The dataset offers temporal diversity, with images dated from the early 20th century to the present, and a global geographic footprint, though with concentration in Europe. Modalities include photographs, hand-drawn sketches, nighttime and daytime scenes, and aerial imagery.

## 4. Ground-Truth Correspondence Format

Each image pair contains a set $\mathcal M$ of ground-truth correspondences:
$$
\mathcal M = \{(x_i^{A},x_i^{B})\}_{i=1}^M
$$
where $x \in \mathbb R^2$ are pixel coordinates in each 640×640 image.

The annotation process yielded a noise $\sigma_{\mathrm{anno}} \approx 1$ px, and evaluation thresholds are set in the range [0, 20] px for symmetric epipolar error. Notably, pairs labeled as "dynamic" (approximately 20) are excluded from F-matrix-based metrics but can be evaluated directly with keypoint-to-keypoint error.

## 5. Evaluation Metrics

HardMatch supports multiple rigorous metrics for benchmarking correspondence algorithms:

**(a) Percentage of Correct Keypoints (PCK):**  
Defined under an epipolar-error threshold $\tau$, given an estimated fundamental matrix $F$, and symmetric epipolar distance $d_{F}(x_i^{A},x_i^{B})$:

$$
\mathrm{PCK}(\tau) = \frac{1}{|\mathcal M|}\sum_{(x_i^{A},x_i^{B})\in\mathcal M} \mathbf{1}[d_{F}(x_i^{A},x_i^{B}) \leq \tau].
$$

**(b) Mean Average Accuracy (mAA):**  
Computed for thresholds up to $D$ pixels (e.g., $D=10$):

Discrete:
$$
\mathrm{mAA}@D = \frac{1}{D}\sum_{\tau=1}^{D}\mathrm{PCK}(\tau)
$$
Continuous:
$$
\mathrm{mAA}@D = \frac{1}{D}\int_{0}^{D}\mathrm{PCK}(t)\,\mathrm dt
$$

**(c) Relative-Pose AUC@θ°:**  
Applicable for datasets like MegaDepth/ScanNet. If $r(\theta)$ is the fraction of image pairs with $\max\{\text{rot-err, ang(angle)-rans}\} \leq \theta^\circ$:

$$
\mathrm{AUC}@\theta = \frac{1}{\theta}\int_0^{\theta}r(\phi)\, d\phi \approx \frac{1}{\theta}\sum_{k=1}^{K}r(\phi_k)\,\Delta\phi
$$

**(d) Image Matching Challenge mAA:**  
Mean Average Accuracy averaged over a specified grid of (rotation, translation) thresholds, balanced per scene.

## 6. Benchmark Performance and Impact

HardMatch provides a clear demarcation between state-of-the-art local feature matching methods. Core results, as reported in [2604.04931], are summarized below (for $N=4,096$ keypoints):

| Method               | mAA@10px (HardMatch, test set) |
|----------------------|-------------------------------|
| ALIKED + LightGlue   | 35.7%                         |
| LoMa-B (ours)        | 51.1%                         |
| LoMa-L (ours)        | 53.5%                         |
| LoMa-G (ours)        | **54.3%**                     |

LoMa-G yields a gain of $54.3\% - 35.7\% = 18.6$ percentage points over ALIKED+LightGlue, and outperforms dense matchers such as RoMa v2 ($\approx 46.5\%$) by approximately 8 percentage points. Per-group analysis shows LoMa-G achieving at least 30% mAA@10px on the most challenging groups (Doppelgängers, Celestial) and approximately 56%–59% on moderate groups (Illumination, Seasonal, Temporal).

Existing standard benchmarks report $>80\%$ PCK@10 px, leading to performance saturation with limited margin for improvement. In contrast, HardMatch's design intentionally preserves significant headroom, facilitating the meaningful evaluation and development of new algorithms.

## 7. Research Significance and Usage

HardMatch addresses limitations of conventional evaluation protocols for local feature matching, which have predominantly relied on image pairs derived from successful 3D reconstructions—typically easy and lacking diversity. By emphasizing deliberately difficult, diverse, and semantically rich pairs, and by employing manual keypoint annotation with low human error, HardMatch enables precise, fine-grained comparative analysis. The availability of well-documented, difficult image pairs encourages benchmarking and advances in robust correspondence estimation, structure-from-motion, and related 3D vision systems [2604.04931].

Source: https://www.emergentmind.com/topics/hardmatch