---
title: 'XoFTR: Cross-modal Feature Matching Transformer'
url: https://www.emergentmind.com/topics/xoftr
type: topic
---

# XoFTR: Cross-modal Feature Matching Transformer

Searching arXiv for the cited XoFTR and SAR–optical registration papers to ground the article in the latest available preprints.
XoFTR, short for **Cross-modal Feature Matching Transformer**, is a detector-free local feature matcher designed for cross-modal, cross-view correspondence estimation between visible and thermal infrared (TIR) images. It extends the LoFTR family with paired masked image modeling pre-training, pseudo-thermal augmentation, a redesigned coarse-to-fine matching pipeline, and sub-pixel refinement, with the explicit aim of handling viewpoint, scale, and texture variation under severe modality gaps. Subsequent work has treated XoFTR as an off-the-shelf baseline for SAR–optical remote-sensing registration, where it serves as a prominent test case for how far visible–thermal cross-modal supervision transfers to a substantially different sensing modality [2404.09692].

## 1. Origin, scope, and problem setting

XoFTR was introduced for **visible–TIR local feature matching** in settings where both **multimodality** and **multiview** variation are present. The underlying problem is to recover correspondences between 2D locations in a visible image \(I^A\) and a TIR image \(I^B\), so that downstream geometric tasks such as relative pose estimation, homography estimation, structure-from-motion, and localization can be performed. The method is detector-free: rather than first detecting sparse keypoints and then describing them, it predicts semi-dense correspondences from deep features and transformer-based contextualization [2404.09692].

The motivation is rooted in the mismatch between sensing physics and geometric variation. Visible imagery records reflected visible light, whereas TIR imagery records emitted thermal radiation in the \(8\text{–}14\,\mu\mathrm{m}\) range. The resulting differences in texture, intensity statistics, field of view, and effective resolution make visible–TIR matching substantially harder than visible–visible matching. XoFTR was proposed in response to the limitations of both handcrafted descriptors and RGB-trained deep matchers, which had difficulty jointly handling non-linear modality differences and large viewpoint or scale changes.

Later remote-sensing studies repositioned XoFTR within a broader taxonomy of matchers. In SAR–optical registration, it is treated as a **deep learning–based, detector-free, transformer-based cross-modal matcher** and is typically evaluated **without architectural changes, fine-tuning, or SAR-specific adaptation**. This reuse is important because it isolates transfer: any observed success or failure can be attributed to the learned representation and inference protocol rather than to task-specific retraining [2502.01002].

## 2. Architecture and matching pipeline

XoFTR follows a **LoFTR-style coarse–fine transformer design**, but its modifications are specifically targeted at cross-modal matching. At test time, the longer side of each image is resized to \(640\) pixels. A ResNet-based CNN backbone extracts multi-scale features at \(1/8\), \(1/4\), and \(1/2\) resolution, denoted \(F^*_{1/8}\), \(F^*_{1/4}\), and \(F^*_{1/2}\) for each modality \(* \in \{A,B\}\) [2404.09692].

The **Coarse-Level Matching Module (CLMM)** operates at \(1/8\) scale. It applies a LoFTR-style transformer with linear self-attention and cross-attention to produce refined coarse features \(\hat{F}_{1/8}^A\) and \(\hat{F}_{1/8}^B\). The coarse similarity matrix is defined as
\[
\mathcal{S}(i,j)=\frac{1}{\tau}\cdot\left\langle \operatorname{Linear}(\hat{F}^{A}_{1/8}(i)), \operatorname{Linear}(\hat{F}^{B}_{1/8}(j))\right\rangle.
\]
Row-wise and column-wise softmax probabilities are then used with an **AdaMatcher-style many-to-one/one-to-many/one-to-one assignment**. This is a decisive departure from the original LoFTR pipeline, because one-to-many coarse assignment is intended to absorb the scale and viewpoint discrepancies that arise when visible and thermal sensors differ substantially in field of view and resolution.

The **Fine-Level Matching Module (FLMM)** refines these coarse correspondences to \(1/2\) scale. It fuses coarse transformer features with backbone features, extracts local windows of sizes \(1\times1\), \(3\times3\), and \(5\times5\) around each coarse match, and processes them hierarchically with self-attention and cross-attention. Fine matching is then performed within local \(5\times5\) windows using dual-softmax probabilities, together with a second confidence threshold \(\theta_f\), yielding one-to-one fine correspondences.

The final stage is the **Sub-pixel Refinement Module (SPRM)**. For each fine match, XoFTR concatenates local features from both modalities and regresses offsets with an MLP followed by \(\tanh\):
\[
\{\delta_{x_A}, \delta_{y_A}, \delta_{x_B}, \delta_{y_B}\} = \operatorname{Tanh}\left(\operatorname{MLP}(\tilde{f}^A_{5\times5}(\hat{i}) \mathbin\Vert \tilde{f}^B_{5\times5}(\hat{j}))\right).
\]
These offsets are added to the discrete \(1/2\)-scale coordinates to produce sub-pixel correspondences. The overall design can be summarized as a cross-modal LoFTR derivative in which coarse many-to-many tolerance, multi-scale local refinement, and sub-pixel regression are all explicitly tied to modality-robust matching.

## 3. Pre-training, augmentation, and optimization

A central feature of XoFTR is that its cross-modal capability is not attributed solely to supervised matching loss. The model first undergoes **paired masked image modeling (MIM)** on the **KAIST Multispectral Pedestrian Detection** dataset, which provides \(95{,}000\) RGB–TIR image pairs. During MIM, images are cropped to the top \(640\times480\) region, \(50\%\) of the image is masked using \(64\times64\) patches, masked locations are replaced by learnable mask tokens at feature scales, and the decoder reconstructs the masked regions using an MSE objective [2404.09692].

The second ingredient is **pseudo-thermal augmentation** during matching fine-tuning on **MegaDepth**. For each RGB pair, the augmentation is applied randomly to one image. The procedure consists of color jitter, grayscale conversion, a randomized cosine transform with parameters \(w_0 = 2\pi/3\), \(w_r = \pi/2\), and \(\theta_r = \pi/2\), followed by random Gaussian blur with a \(5\times5\) kernel. The purpose is to generate thermal-like intensity transformations without relying on learned image translation, thereby exposing the matcher to broad modality-like variation.

Matching fine-tuning is supervised by three losses: a coarse-level focal loss \(\mathcal{L}_c\), a fine-level focal loss \(\mathcal{L}_f\), and a sub-pixel symmetric epipolar loss \(\mathcal{L}_{sub}\). The total loss is
\[
\mathcal{L}_{total} = \lambda_c \mathcal{L}_c  + \lambda_f\mathcal{L}_f + \lambda_{sub}\mathcal{L}_{sub},
\]
with \(\lambda_c = 0.5\), \(\lambda_f = 0.3\), and \(\lambda_{sub} = 10^4\). Coarse and fine supervision are generated from depth maps and camera poses, while the sub-pixel term uses the essential matrix. The reported training protocol uses Adam, \(2.5\times10^{-4}\) learning rate for MIM pre-training over \(9\) epochs with batch size \(2\), and \(2\times10^{-3}\) learning rate for fine-tuning with batch size \(16\). The coarse and fine matching thresholds are \(\theta_c = 0.3\) and \(\theta_f = 0.1\).

## 4. Performance on visible–thermal benchmarks

XoFTR was evaluated on the **METU-VisTIR** benchmark, a visible–thermal dataset collected with a DJI Mavic 3 Thermal drone. The thermal camera has resolution \(640\times512\), field of view \(61^\circ\), and spectrum \(8\text{–}14\,\mu\mathrm{m}\); the RGB camera has resolution \(3840\times2160\) and field of view \(84^\circ\). The benchmark contains six outdoor scenes, with separate **cloudy–cloudy** and **cloudy–sunny** splits of \(1382\) and \(1208\) image pairs respectively [2404.09692].

| Benchmark | Metric | XoFTR result |
|---|---:|---:|
| METU-VisTIR cloudy–cloudy | AUC@20° | 55.06 |
| METU-VisTIR cloudy–sunny | AUC@20° | 45.03 |
| LGHD + FusionDN homography benchmark | AUC@20px | 48.15 |

On relative pose estimation, XoFTR substantially outperformed the listed baselines on both METU-VisTIR splits. In cloudy–cloudy, its AUC@20° is **55.06**, compared with **19.17** for DKM and **14.11** for LoFTR. In cloudy–sunny, where illumination changes compound the modality gap, XoFTR reaches **45.03** AUC@20°, again well ahead of DKM at **23.60** and LoFTR at **16.36**. On homography estimation over the combined **LGHD LWIR/RGB** and **FusionDN RoadScene** benchmark with synthetic homographies, XoFTR reaches **48.15** AUC@20px, exceeding LoFTR at **30.23** and ASpanFormer at **36.42** at that threshold.

The ablation study identifies the principal contributors to this performance. On the cloudy–sunny split, removing MIM pre-training lowers AUC@20° from **45.03** to **42.93**. Removing pseudo-thermal augmentation causes a much larger drop, from **45.03** to **14.94**. Replacing one-to-many coarse assignment with one-to-one reduces AUC@20° to **38.73**, and reverting to LoFTR’s original coarse-to-fine design reduces it to **26.77**. Removing sub-pixel refinement changes AUC@20° from **45.03** to **44.68**, indicating a modest but consistent benefit, while removing the fine-level threshold or positional bias yields **43.56** and **43.36** respectively. Runtime on an A5000 at \(640\times512\) is reported as approximately **116 ms**, compared with **102 ms** for LoFTR.

## 5. Transfer to SAR–optical satellite registration

The main cross-domain question raised in later work is whether a matcher trained for visible–thermal invariance also transfers to **SAR–optical satellite registration**. In a 24-family zero-shot evaluation on SpaceNet9 and two additional SAR–optical benchmarks, XoFTR is the only matcher in the sweep described as being **explicitly trained for cross-modal visible–thermal matching**. It is grouped with other cross-modal or multimodal-pretrained methods such as MINIMA-XoFTR, MINIMA-RoMa, and MatchAnything-ELoFTR [2604.10217].

Under the main SpaceNet9 protocol—**affine** geometry via OpenCV `estimateAffine2D`, **512 px** tiles with **256 px** overlap, maximum long side **1024 px**, RANSAC threshold **3.0 px**, minimum **4** inliers, and **percentile normalization** for XoFTR—its performance is among the best reported: **3.0 px** mean tie-point error, **78.4%** Success@5, **90.5%** Success@10, and **0.00** failure rate. This mean error is tied with **RoMa**, which also attains **3.0 px** but does so without cross-modal training. **MatchAnything-ELoFTR** follows at **3.4 px**, while **LoFTR** records **5.1 px** mean error. The resulting comparison is central to the paper’s notion of **asymmetric transfer**: explicit cross-modal supervision helps, but it does not uniformly dominate strong single-modality or synthetic cross-modal alternatives.

Protocol sensitivity is a major part of the interpretation. The four-stage evaluation pipeline includes preprocessing, tiled correspondence extraction, geometric filtering, and transform-based error computation on tie points. Across ablations, geometry model, tile size, overlap, RANSAC threshold, and inlier gating can move accuracy substantially. In the broader sweep, **affine geometry alone reduces mean error from \(12.34\) to \(9.74\) px**, reinforcing that deployment choices can rival or exceed matcher-to-matcher differences. XoFTR is reported as comparatively stable under threshold sweeps and tiled affine fitting: it is less fragile than repurposed 3D-reconstruction matchers such as MASt3R and DUSt3R, yet still materially affected by protocol design. On SRIF and SARptical, XoFTR remains competitive but is not highlighted as the leading method.

## 6. Multi-resolution SAR–optical evidence, limitations, and research directions

A distinct picture emerges from the **MultiResSAR** benchmark, a dataset of **10,850** SAR–optical pairs spanning Sentinel-1, HT1-A, GF-3, and Umbra data, with resolutions from **10 m** down to **0.16 m** and scenes including urban, rural, plains, hills, mountains, and water. Here XoFTR is evaluated as an off-the-shelf deep matcher using public code and author-recommended parameters, again without SAR-specific redesign or fine-tuning [2502.01002].

On the full benchmark, XoFTR is reported as the **best deep learning method** by two of the four aggregate metrics: **Success Rate (SR) 40.58%** and **RMSE 3.03 pixels**, together with **NCM 244.26** and **TM 0.032 s** per pair. These numbers place it ahead of other deep baselines such as RoMa (**SR 35.26%**, RMSE **3.15**) and XFeat (**SR 36.29%**, RMSE **4.94**). At the same time, the traditional **RIFT** method is more robust in aggregate, with **SR 66.51%**, though at higher RMSE (**3.58 px**) and much greater runtime (**5.283 s**). The resulting contrast is precise: XoFTR is fast and geometrically accurate when successful, but its cross-scene robustness remains limited relative to the strongest handcrafted SAR-aware baseline.

The most severe limitation appears at **sub-meter SAR resolution**. MultiResSAR includes an **Umbra–GE** subset of **850** pairs at **0.16 m** SAR resolution. The study states that on sub-meter image pairs, nearly all algorithms fail, and in the four illustrated groups **only RoMa** achieves correct registration. XoFTR is therefore part of the broader failure mode in ultra-high-resolution SAR, where fine speckle structure, layover, shadow, and strong 3D effects break the assumptions of 2D cross-modal matching.

These results argue against a common simplification that cross-modal pretraining alone should solve remote-sensing transfer. A more defensible reading is that XoFTR validates the utility of explicit cross-modal supervision—particularly through its improvement over LoFTR and its strong SpaceNet9 results—while also revealing that robustness on SAR–optical data depends on additional factors: sensing-physics mismatch, geometric verification, dataset coverage, and scale. The cited studies converge on several open directions: **noise suppression**, **SAR-aware feature learning**, **3D geometric fusion**, **cross-view transformation modeling**, and **domain-specific deep learning optimization** such as fine-tuning on datasets like MultiResSAR or SpaceNet9. Within that landscape, XoFTR occupies a specific position: a strong cross-modal transformer baseline whose design transfers surprisingly well beyond visible–thermal matching, but whose limitations remain pronounced in the hardest SAR regimes [2502.01002].

Source: https://www.emergentmind.com/topics/xoftr