XRefine: Attention-Guided Keypoint Match Refinement

Abstract: Sparse keypoint matching is crucial for 3D vision tasks, yet current keypoint detectors often produce spatially inaccurate matches. Existing refinement methods mitigate this issue through alignment of matched keypoint locations, but they are typically detector-specific, requiring retraining for each keypoint detector. We introduce XRefine, a novel, detector-agnostic approach for sub-pixel keypoint refinement that operates solely on image patches centered at matched keypoints. Our cross-attention-based architecture learns to predict refined keypoint coordinates without relying on internal detector representations, enabling generalization across detectors. Furthermore, XRefine can be extended to handle multi-view feature tracks. Experiments on MegaDepth, KITTI, and ScanNet demonstrate that the approach consistently improves geometric estimation accuracy, achieving superior performance compared to existing refinement methods while maintaining runtime efficiency. Our code and trained models can be found at https://github.com/boschresearch/xrefine.

• The paper introduces a detector-agnostic model using cross-attention on raw 11×11 patches to achieve sub-pixel keypoint refinement.
• It significantly improves pose estimation accuracy with up to 15.99% AUC5 gains on MegaDepth while maintaining minimal runtime overhead.
• The approach generalizes across detectors and extends to multi-view track refinement, enhancing modularity in 3D vision systems.

XRefine: Attention-Guided Keypoint Match Refinement — An Expert Analysis

Motivation and Problem Landscape

Sparse keypoint detection and matching underpin core pipelines in 3D computer vision, notably in SfM, SLAM, and visual localization. Although advances in learning-based detectors (e.g., SuperPoint, ALIKED) boost repeatability and robustness, these detectors frequently suffer from limited spatial accuracy of matched keypoints, especially since they operate on images independently. This misalignment at the sub-pixel level directly degrades the precision of downstream geometric tasks, including relative pose estimation and triangulation.

Existing approaches to sub-pixel refinement—such as Keypt2Subpx and PixSfM—show improvements but are inherently tied to detector-specific representations (requiring score maps or descriptors) or involve computationally expensive optimization schemes, hampering general applicability and efficiency.

XRefine: Detector-Agnostic Refinement via Cross-Attention

XRefine introduces a detector-agnostic architecture for sub-pixel match refinement. It operates exclusively on raw image patches centered at matched keypoints, without employing detector features or scores. This enables the training of a single general model that transfers across a wide landscape of detectors, classical or learned.

The core architectural mechanism is cross-attention between patch embeddings from the two input views. Given $11{\times}11$ grayscale patches extracted at each (putatively matched) keypoint, an encoder yields a compact embedding for each patch (down to $3{\times}3$ per patch). Cross-attention then allows the model to explicitly reason about geometric and photometric correspondence using both local context and direct information passing between views. A convolutional "score map head" followed by spatial soft-argmax predicts the sub-pixel offset for each keypoint.

Figure 2: XRefine's architecture leverages cross-attention on compact CNN-derived embeddings from paired patches to efficiently and jointly refine keypoint locations.

Importantly, XRefine provides both a general model (trained agnostic to the detector and applicable out-of-the-box) and a specific model (fine-tuned to a detector for potentially marginal accuracy gains).

Generalization to Multi-View Track Refinement

Many 3D pipelines require consistent multi-view, rather than pairwise, refinement. XRefine introduces an $n$-view variant: fixes one keypoint as a reference and only refines the locations of the others in the track, propagating consistency while retaining pairwise efficiency (linear in track length $T$).

Empirical Evaluation

Relative Pose Estimation

Comprehensive experiments on MegaDepth, ScanNet, and KITTI demonstrate consistent accuracy improvements over existing match refinement baselines (Keypt2Subpx, PixSfM), with higher AUC5 gains and minimal latency overhead. For instance, on MegaDepth, XRefine general and specific models achieve up to 15.99% average improvement in AUC5 for pose estimation, outperforming Keypt2Subpx (8.07%) and PixSfM (11.92%).

Figure 3: XRefine delivers improved pose estimation (AUC5 gains) with runtime efficiency, outperforming Keypt2Subpx and PixSfM across several detectors and datasets.

Crucially, the general model produces substantial gains even when operating with SIFT, SuperPoint, and ALIKED, confirming robust cross-detector generalization.

Effect of Keypoint Accuracy on Downstream Tasks

Controlled perturbation analyses demonstrate that sub-pixel inaccuracies degrade AUC5 pose estimation scores nonlinearly—small increases in localization error lead to considerable performance drops.

Figure 4: The impact of keypoint localization error on pose estimation: as Gaussian noise increases, pose accuracy (AUC5) rapidly declines.

Qualitative and Comparative Analysis

Visualization on MegaDepth illustrates qualitative improvements over baselines. Refined keypoints (yellow dots) cluster more tightly around true correspondences after applying XRefine.

Figure 1: XRefine-corrected keypoints (yellow) demonstrate superior localization consistency versus initial SuperPoint matches (red), using only the inner $11\times11$ patch.

Further, comprehensive comparisons with Keypt2Subpx and PixSfM reveal that XRefine not only surpasses them quantitatively but also reduces detector- and dataset-specific variability.

Figure 5: XRefine consistently improves keypoint locations over Keypt2Subpx and PixSfM across diverse patch examples, yielding sharper clustering on true correspondences.

Runtime and Scalability

XRefine refines 2048 matches in only 3.61 ms on an RTX A5000 GPU, nearly matching Keypt2Subpx, while vastly outperforming PixSfM (70.28 ms without, 1435 ms with full S2DNet inference).

Multi-View Triangulation

The $n$-view extension is validated on ETH3D, yielding higher triangulation accuracy across both indoor and outdoor sequences, though joint global optimization (as in PixSfM) remains superior in accuracy at a substantial computational cost.

Implications and Future Directions

The introduction of a detector-agnostic, cross-attention-based match refinement mechanism signals a shift toward modular, interoperable keypoint pipelines that can more flexibly leverage advances in both classical and learned detectors. The computational efficiency demonstrated by XRefine underscores its suitability for resource-constrained deployment at scale and on edge AI devices.

Future research could extend XRefine's cross-attention mechanism to operate directly on sets of $n$ patches for truly global track consistency, potentially bridging the gap with globally optimal but slow optimization-based frameworks like PixSfM. Further avenues include adaptation to non-rigid matching contexts, the incorporation of context from higher-resolution patches, and joint learning with upstream detector/matcher modules for cohesive end-to-end optimization.

Conclusion

XRefine advances the state of the art in match refinement by providing an efficient, detector-agnostic, cross-attention-based model for sub-pixel keypoint adjustment. Extensive experiments show strong and robust improvements in geometric accuracy across detectors and datasets, without trade-offs in runtime. Its architectural flexibility and empirical effectiveness position it as a critical building block for future high-precision, modular 3D vision systems.