- The paper introduces TAR, a coarse-to-fine registration framework that combines visual features with RemoteCLIP text semantics to reduce correspondence ambiguity between optical and SAR images.
- TAR achieves the highest CMR@1 on SEN1-2 and OSdataset, reaching 80.3% and 59.6%, respectively, while remaining less precise in RMSE than XoFTR.
- The ablations show that semantic text features improve coarse matching but harm fine localization, while expanding the prompt library from 37 to 224 categories raises SEN1-2 CMR@1 from 78.1% to 80.3%.
Motivation and problem statement
Optical and SAR image registration must establish spatial correspondences between modalities whose imaging mechanisms produce substantial differences in intensity, texture, and local structure. Existing deep learning approaches—whether CNN-based correlation learners or Transformer-based dense matchers such as LoFTR and XoFTR—rely exclusively on visual features. The authors argue that this is insufficient when cross-modal appearance discrepancies coincide with large geometric deformations: the model must simultaneously maintain feature invariance across modalities and discrimination across local patterns while searching over a wide spatial range, which increases matching ambiguity. TAR addresses this by injecting remote sensing text semantic priors into the visual pipeline, exploiting the observation that optical and SAR images of the same region share consistent scene- and land-cover-level semantics even when their appearance diverges.
Framework architecture
TAR comprises three modules. The Multi-Scale Visual Feature Learning (MSFL) module uses a ResNet backbone with a feature pyramid to extract low-level features at H/2×W/2 resolution (for fine matching) and high-level features at H/8×W/8 (for coarse matching) from each modality independently.
The Text-Assisted Feature Enhancement (TAFE) module is the core contribution. A prompt set is constructed from 224 remote sensing categories—basic scenes and land-cover objects expanded via GPT and filtered—with each category rendered as "a satellite image of [category]" and encoded by a frozen RemoteCLIP text encoder. High-level visual features are enhanced through two parallel branches:
- Visual-text interaction: cross-attention between visual tokens and the text embedding library, providing modality-agnostic semantic references.
- Visual-visual interaction: self-attention within each modality followed by cross-attention across modalities, capturing intra-image structure and cross-modal correlations.
The two branches are concatenated channel-wise and fused by a three-layer MLP. Notably, text interaction is applied only to high-level features; the authors explicitly justify this design choice by arguing that low-level features encode local structures and edges where global semantic injection would disturb fine localization. This claim is empirically validated in the ablation study.
The Coarse-to-Fine Dense Matching (CFDM) module follows the LoFTR paradigm: coarse matching computes a dual-softmax confidence matrix over enhanced high-level features with mutual nearest-neighbor filtering, then fine matching crops 3×3 local windows around coarse correspondences, applies self- and cross-attention on low-level features, and regresses local offsets. Training combines a focal loss for coarse matching (addressing positive/negative imbalance) with an uncertainty-weighted L2 loss for fine matching, where per-sample weights are derived from the standard deviation of local feature similarity—an explicit mechanism to down-weight unreliable correspondences during optimization.
Experimental results
Experiments are conducted on SEN1-2 (10 m resolution, 12,642 training / 1,000 test pairs) and OSdataset (1 m GF-3 spotlight SAR versus Google Earth optical, 6,297 / 1,000 pairs), with synthetic affine perturbations applied to SAR images (scale [0.7,1.3], rotation ±35°, translation up to 10% of image size). Key results:
| Method |
SEN1-2 RMSE |
SEN1-2 CMR@1 |
OSdataset RMSE |
OSdataset CMR@1 |
| LoFTR |
2.43 |
62.8 |
2.81 |
52.2 |
| GDROS |
1.78 |
47.5 |
2.16 |
42.6 |
| XoFTR |
1.12 |
59.4 |
1.57 |
32.5 |
| TAR |
1.68 |
80.3 |
2.03 |
59.6 |
TAR achieves the best CMR under all thresholds (τ=1,3,5 pixels) on both datasets. The CMR@1 gains are pronounced: +17.5% over LoFTR, +32.8% over GDROS, and +20.9% over XoFTR on SEN1-2. Traditional methods (RIFT, LNIFT, OS-Flow) fail almost entirely under these perturbations, with CMR@1 of 0.0.
A notable and candidly discussed result is that TAR does not achieve the best RMSE—XoFTR attains lower RMSE on both datasets (1.12 vs. 1.68 on SEN1-2). The authors attribute this to RMSE's sensitivity to a small number of large-residual samples, whereas CMR reflects registration success rate. This is a fair characterization but also a limitation: TAR improves matching stability more than average localization precision.
Ablation findings
Two ablations substantiate the design decisions. First, inserting visual-text interaction into the fine matching stage degrades performance severely (CMR@1 drops to 49.7% versus 80.3% for coarse-only insertion), confirming that global scene semantics are incompatible with sub-pixel localization. Second, expanding the text vocabulary from 37 basic categories to 224 GPT-expanded categories further improves CMR@1 from 78.1% to 80.3%, indicating that richer semantic coverage aids coarse correspondence disambiguation. Qualitative results show TAR producing denser, more uniformly distributed correspondences than all baselines on challenging samples.
Limitations and open questions
Several constraints should be noted. The evaluation relies on synthetically generated affine perturbations rather than real geometric misalignments between independently acquired optical-SAR pairs, so performance under true non-rigid deformations (e.g., SAR layover, foreshortening) remains unverified. The framework depends on RemoteCLIP's text encoder quality and on a fixed prompt template; the sensitivity of results to prompt engineering and to the frozen encoder choice is not analyzed. The text library covers generic land-cover categories, leaving open how the approach behaves on scenes poorly represented in the 224-category vocabulary. Finally, the RMSE gap relative to XoFTR suggests that text priors improve correspondence reliability but do not directly refine localization accuracy—a question the paper leaves open regarding whether finer-grained, object-level text supervision could benefit the fine matching stage as well.
Conclusion
TAR demonstrates that remote sensing text semantic priors, injected via cross-attention into high-level visual features only, materially improve the success rate of optical–SAR registration under combined appearance discrepancy and geometric deformation. The central empirical finding—that CMR@1 improves by double-digit margins over strong Transformer-based baselines while RMSE remains competitive—supports the paper's thesis that modality-invariant semantic references reduce ambiguity in wide-range coarse matching. The main open issues are validation under realistic non-rigid deformations and the dependence of gains on the specific vision-language foundation model and prompt construction.