Cross-Modality Ship Re-ID

• CMS Re-ID is a remote sensing task that matches optical and SAR ship images by addressing significant modality differences.
• Recent methods employ transformer architectures with contrastive pretraining and dual-head tokenizers for effective feature alignment.
• Advanced techniques such as generative augmentation and feature-space modulation significantly enhance retrieval accuracy and robustness.

Cross-Modality Ship Re-Identification (CMS Re-ID) is the remote sensing task of matching instances of the same physical ship across different sensing modalities, notably optical (visible spectrum) and synthetic aperture radar (SAR) imagery. This problem is central for persistent, all-weather maritime target tracking, enabling forensic investigation, shipping analysis, and law enforcement in variable observation conditions. The principal challenges stem from the substantial modality gap—differences in imaging physics, data representation, and statistical properties—which prohibit naive feature-space matching and require sophisticated alignment strategies to achieve modality-invariant ship descriptors (Wang et al., 27 Jun 2025, Zhao et al., 3 Dec 2025, Xian et al., 24 Dec 2025).

1. Formal Problem Definition and Motivations

Given a probe image of a ship from one modality (optical or SAR), CMS Re-ID aims to retrieve, from a gallery of images in the other modality, all instances corresponding to the same ship. Precise notation for the task is as follows: Let $x^O \in \mathbb{R}^{H \times W \times 3}$ denote a cropped optical image, $x^S \in \mathbb{R}^{H \times W}$ a SAR crop, and $\varphi: x \rightarrow f \in \mathbb{R}^D$ an embedding function. The objective is to enforce

$\|\varphi(x^O_i) - \varphi(x^S_j)\|_2$

to be small for matching (same-identity) pairs and large otherwise, across substantial domain gaps (Wang et al., 27 Jun 2025).

Motivating factors include the complementary operational properties of the imaging modalities: optical imagery offers higher spatial resolution but is susceptible to clouds and darkness, while SAR provides robust all-weather, day-night operation but exhibits distinct scattering physics, leading to different visual appearance and challenges such as speckle noise and low scene texture (Zhao et al., 3 Dec 2025).

2. Datasets and Evaluation Protocols

The Hybrid Optical and Synthetic Aperture Radar Ship Re-Identification (HOSS ReID) dataset is the first publicly available benchmark for this task. It contains 1,065 optical and 767 SAR images (1,832 total) derived from 449 unique ships and 163 gallery distractors, with imaging sequences from low-Earth orbit satellites (Jilin-1, TY-MINISAR) under diverse geometry and weather. Annotations are provided as bounding-box crops with Market1501-style identities, and rigorous geometric/radiometric corrections—except for DEM orthorectification—are applied (Wang et al., 27 Jun 2025).

Typical evaluation splits comprise:

• Training: 574 optical + 489 SAR crops (361 identities)
• Query: 88 identities $\times$ 2 modalities (176 images)
• Gallery: 88 identities + 163 distractors (593 images)

CMS Re-ID is evaluated using mean Average Precision (mAP), and Cumulative Matching Characteristic (CMC) Rank-1/5/10. Evaluation protocols include ALL→ALL (queries and gallery include both modalities), Optical→SAR, and SAR→Optical (Zhao et al., 3 Dec 2025):

Protocol	Queries	Gallery
ALL→ALL	176 (88 O+S)	593 (O+S+dis.)
Optical→SAR	65 O	190 SAR
SAR→Optical	67 SAR	403 Optical

3. Baseline and Advanced Methodologies

Early approaches leverage conventional deep architectures with explicit alignment components. TransOSS (Wang et al., 27 Jun 2025), a transformer-based baseline, refines patch embeddings separately for optical and SAR patches via a dual-head tokenizer, adds positional and modality information embeddings, and incorporates ship-size metadata. Two-stage training—contrastive pretraining with large-scale optical–SAR pairs, followed by supervised fine-tuning with triplet and ID (cross-entropy) losses—establishes a modality-aligned embedding space.

Table: Core Algorithmic Features of Notable Methods

Method	Alignment Mechanism	Backbone	Pretraining / Augmentations
TransOSS	Dual-head, MIE, SSE, CL	ViT-base	Contrastive learning on paired O/SAR
DRI	Feature-space modulation	Frozen VFM (ViT)	OE + Modulators, only ~1.5–7M parameters
MOS	Distribution, Generative	ViT-base	MCRL, BBDM synthesis + inference fusion

Explanations:

• TransOSS introduces a dual-head tokenizer and auxiliary embeddings for modality and ship size, achieving SOTA at 57.4% mAP, 65.9% R1 on HOSS. Ablations indicate that contrastive pretraining is the dominant source of improvement in bridging the modality gap (Wang et al., 27 Jun 2025).
• Domain Representation Injection (DRI) (Xian et al., 24 Dec 2025) utilizes a fully frozen Vision Foundation Model (e.g., DINOv3-ViT), and injects learned domain-specific offsets into the intermediate features via lightweight Offset Encoder and Modulator structures. DRI achieves 57.9% mAP (ViT-S, 1.54M params) and 60.5% mAP (ViT-L, 7.05M params), outperforming TransOSS with an order-of-magnitude reduction in trainable parameters. DRI's injection strategy yields robust improvements in cross-modal generalization.
• MOS (Zhao et al., 3 Dec 2025) applies a two-stage mechanism: Modality-Consistent Representation Learning (MCRL) explicitly denoises SAR inputs and enforces class-wise feature distribution alignment via a 2-Wasserstein loss, while Cross-modal Data Generation & Feature Fusion (CDGF) synthesizes pseudo-SAR samples from optical inputs using a Brownian bridge diffusion model. MOS surpasses prior SOTA with mAP=60.4%, R1=68.8% (ALL→ALL), and especially improves SAR→Optical retrieval (>16 R1 points).

4. Modality Gap Mitigation Strategies

Mitigating the optical–SAR domain gap is central to CMS Re-ID. Notable strategies include:

• Representation-level Alignment: MCRL (Zhao et al., 3 Dec 2025) minimizes intra-class Wasserstein distances between optical and SAR feature distributions, explicitly regularizing first and second moments (means, variances) of the two modalities. This shifts feature clustering from modality-centric to identity-centric organization.
• Augmenting Feature Spaces via Generation: CDGF in MOS uses a diffusion model (Brownian bridge) to generate modality-translated (e.g., SAR from optical) samples, whose features are fused with real features using a normalized convex combination at inference. This late-stage fusion leverages synthetic cross-modal cues to mitigate residual domain discrepancy.
• Feature-Space Domain Injection: DRI (Xian et al., 24 Dec 2025) introduces post-normalization, per-block feature modulations—learned from a compact Offset Encoder—so that the frozen backbone's representations are locally steered toward domain-invariant manifolds. Empirical ablations suggest that both post-norm positioning and zero initialization of the modulator are essential; deviations harm performance substantially.

SAR denoising techniques, such as percentile-based value truncation and linear rescaling, are systematically applied to reduce coherent speckle prior to feature extraction (Zhao et al., 3 Dec 2025).

5. Quantitative and Ablative Findings

The main recent results on HOSS ReID are summarized:

Method	mAP (ALL)	R1 (ALL)	mAP O→S	R1 O→S	mAP S→O	R1 S→O
TransOSS (ViT-base)	57.4	65.9	48.9	33.8	38.7	29.9
MOS (ours)	60.4	68.8	51.4	40.0	48.7	46.3
DRI (ViT-S, 1.5M)	57.9	67.0	--	--	--	--
DRI (ViT-L, 7M)	60.5	69.9	55.6	--	45.0	--

Ablation studies reveal:

• For MOS, MCRL alone provides +1.9% mAP and +2.3% R1 (ALL→ALL) with especially marked gains in SAR→Optical. CDGF alone is effective only if the feature space is well aligned; their combination yields maximal improvement (Zhao et al., 3 Dec 2025).
• For DRI, ablations over injection site (post-norm is critical), modulator design (linear zero-init optimal), and OE depth/dimension provide detailed sensitivity profiles (Xian et al., 24 Dec 2025).
• Auxiliary signals (e.g., modality and size embeddings in TransOSS) incrementally improve baseline transformer performance but are outperformed by feature- or distribution-level alignment schemes (Wang et al., 27 Jun 2025, Xian et al., 24 Dec 2025, Zhao et al., 3 Dec 2025).

6. Limitations and Future Directions

Current CMS Re-ID benchmarks are constrained by dataset scale and diversity; geographic, seasonal, and vessel-type coverage remain limited (Wang et al., 27 Jun 2025). Geolocation noise arises from the absence of DEM-based ortho-rectification. Synthetic sample realism in generative alignment frameworks (e.g., CDGF) governs attainable performance, with plausible failure if distributions are not faithfully bridged (Zhao et al., 3 Dec 2025).

Future research directions include:

• Self-supervised or unsupervised domain adaptation to leverage unlabeled SAR data,
• Extending to more modalities (multispectral, hyperspectral, textual),
• Employing satellite metadata for spatio-temporal context embedding,
• Lightweight, on-the-fly generative augmentation or joint end-to-end training of generation and retrieval modules (Wang et al., 27 Jun 2025, Zhao et al., 3 Dec 2025).

These proposed strategies aim to address dataset scale, domain variety, and the practical constraints of cross-modality tracking in operational environments.

7. Significance and Impact

CMS Re-ID represents a frontier problem at the confluence of multi-modal representation learning, maritime surveillance, and remote sensing. The rapid progress from ViT baselines and dual-head tokenizers to advanced methods such as MOS and DRI demonstrates the effectiveness of explicit distributional, generative, and feature-space adaptation mechanisms for bridging profound domain gaps between optical and SAR imagery. The SOTA performance margins, approaching or surpassing 60% mAP and ~70% R1 on challenging protocols, indicate robust progress, yet significant headroom remains for generalization, scalability, and deployment under real-world conditions (Wang et al., 27 Jun 2025, Xian et al., 24 Dec 2025, Zhao et al., 3 Dec 2025).