SkySense V2: Unified Multi-Modal Remote Sensing

• SkySense V2 is a unified multi-modal remote sensing foundation model that fuses high-resolution optical, multi-spectral, and SAR data using a transformer-based architecture.
• It leverages adaptive patch merging, learnable modality prompt tokens, and a mixture of experts to enhance parameter efficiency and cross-modal feature alignment.
• Empirical results across 16 datasets demonstrate that SkySense V2 outperforms its predecessor by an average of 1.8 points in various Earth observation tasks.

SkySense V2 is a unified multi-modal remote sensing foundation model (MM-RSFM) designed for versatile Earth observation tasks such as urban planning, environmental monitoring, and disaster management. It advances prior approaches by consolidating multiple sensor modalities—high-resolution (HR) optical, medium-resolution multi-spectral (MS), and synthetic aperture radar (SAR)—within a single transformer-based architecture, augmenting generalization, parameter efficiency, and cross-modal representation fidelity. This design leverages several innovations: adaptive patch merging, learnable modality prompt tokens, and a large-scale mixture of experts (MoE) module, supported by a bundle of remote sensing-tailored self-supervised learning objectives. Empirical evaluation across 16 datasets and seven task categories demonstrates that SkySense V2 outperforms its predecessor and other baselines by an average of 1.8 points, particularly excelling in multi-modal and out-of-domain generalization scenarios (Zhang et al., 18 Jul 2025).

1. Unified Transformer-Based Architecture

SkySense V2 employs a single hierarchical transformer backbone, supporting HR optical, MS, and SAR data jointly. The architecture comprises four sequential stages:

• Stages 1–2: Swin Transformer V2 blocks with windowed self-attention, concentrating on high-resolution, localized features.
• Stages 3–4: Global Vision Transformer blocks, aggregating context at coarser spatial resolutions.

All backbone parameters are modality-shared except the initial patch-embedding “tokenizer” layers, which remain modality-specific. This design achieves both parameter efficiency and cross-modal knowledge transfer, in contrast to the three-backbone scheme used in the original SkySense.

2. Self-Supervised Learning Framework

SkySense V2 is pre-trained using a teacher–student self-supervised paradigm, drawing from but extending DINOv2 for the remote sensing domain. The objective composes three synergistic components:

• Multi-Granularity Contrastive Learning (MGCL): Contrastive objectives are applied at the pixel, object, and global image levels across and within modalities, as well as on their geospatially fused feature space.
• Unsupervised Geo-Context Prototype Learning (GCPL): Latent features are clustered by geographic region, with each feature vector encouraged to align with its region’s prototype, reinforcing spatial context.
• Dense Image-Text Alignment (ITA): Each pixel’s vision embedding is aligned with corresponding OpenStreetMap land-use category labels via a CLIP text encoder, imposing semantic consistency.

A novel Query-based Semantic Aggregation Contrastive Learning (QSACL) module introduces learnable queries that aggregate semantically consistent regions from various augmentations before applying the contrastive loss. The QSACL loss is formally: $$\mathcal{L}_{QSACL} = \frac1{2m}\sum_{i=1}^m\Big[\, \mathcal{L}_{CL}(z_i^g,\,z_i^{l\prime}) +\mathcal{L}_{CL}(z_i^l,\,z_i^{g\prime}) \Big]$$ where $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$, and $\mathcal{H},\mathcal{H}'$ are projection heads for student and teacher.

$$\mathcal{L} = \lambda_1\mathcal{L}_{MGCL} + \lambda_2\mathcal{L}_{ITA} + \lambda_3\mathcal{L}_{QSACL}$$

with all weights set to unity during pre-training.

3. Adaptive Patch Merging and Modality Prompting

Adaptive Patch Merging (APM)

Given the significant variation in ground sample distance among modalities, APM modules sit after each stage (excluding stage 1), downsampling spatial tokens while respecting alignment:

• For optical: 2×2 neighboring tokens (size $c$) are concatenated (yielding $4c$), linearly mapped to $2c$, reducing spatial resolution by 4×.
• For MS/SAR: APM applies linear projection, maintaining spatial resolution.

Learnable Modality Prompt Tokens

To preserve modality-specific features after shared backbone stages, \emph{N} learnable prompt tokens per modality are injected at stages 3 and 4. The process is

$$\left[ E_{drop},\,E_i^j \right] = \mathcal{F}_{j-1}([P_i^{j-1}, E_i^{j-1}])$$

where $E_i^j$ is the stage input, $P_i^j$ the prompt, and $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$0 the block function. Prompts are concatenated then dropped after each respective block.

This configuration maintains parameter sharing while restoring flexibility to model per-modality statistical structure.

4. Mixture of Experts and Optimization

SkySense V2 incorporates a sparse MoE module in the last six blocks of the backbone. With $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$1 experts and a top-1 gating scheme, each token is routed through its highest-weighted expert; the MoE computation is: $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$2 where $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$3 is the gating function, $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$4 is the $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$5th FFN expert, and $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$6 are top-k indices. This approach nearly triples parameter count but maintains FLOPs efficiency by its sparsity.

Training uses AdamW optimizer (batch 1024, 128 H20 GPUs, 44 500 H20-GPU-hours), with cosine LR decay and auxiliary MoE routing loss. EMA teacher momentum is ramped from 0.996 to 1.0.

5. Empirical Evaluation and Ablation Results

Evaluation covers seven categories and sixteen benchmarks, including static/temporal classification, object detection, semantic segmentation, change detection, and multi-modal tasks:

Task Type	Representative Datasets	Metrics / Gains
Scene Classification	AID, RESISC-45	OA: +0.7 to +1.5
Object Detection	DIOR, DIOR-R, FAIR1M	mAP@50: +0.8 to +1.4
Semantic Segmentation	iSAID, Potsdam, Dyna-Pla	mIoU: +1.5
Change Detection	LEVIR-CD, OSCD, Dyna-S2	F1: +2.7, OSCD +5.2
Multi-Modal Segmentation	Dyna-MM, PASTIS-MM	mIoU/OA: +1.1 to +1.5
Multi-Modal Scene Cls.	BigEarthNet-MM (S2+S1)	mAP: +1.6

Ablation studies indicate that:

• Transitioning from windowed to global attention, adding modality prompt tokens, MoE, and QSACL each synergistically increase representation quality.
• Modality prompts boost feature separability (t-SNE visualization), especially crucial for multi-modal downstream transfer.
• Preserving higher spatial resolution via selective APM boosts segmentation accuracy.
• MoE effectiveness depends on unfreezing routing gates in fine-tuning; randomization or freezing reverts to baseline performance.
• Increasing QSACL query count saturates at $$\mathcal{L}_{CL}(x,x') =-\,\mathcal{H}(x)\,\log\bigl(\mathcal{H}'(x')\bigr)$$7, ensuring semantic variability is fully captured.

SkySense V2 sustains robust performance on out-of-domain sensors, improving average mIoU by +1.8 over its predecessor.

6. Limitations and Prospective Enhancements

Despite its broad applicability, SkySense V2’s current instantiation does not natively accommodate natural language or geographic knowledge representations beyond OSM, and omits modalities such as hyperspectral or LiDAR. Joint vision–language pre-training, hierarchical or larger-scale MoE capacity, and domain adaptation for more exotic input types are identified as priorities for subsequent versions. A plausible implication is that extending SkySense V2 with geo-knowledge graphs and universal vision-language representations may be required to approach universality in remote sensing foundation models (Zhang et al., 18 Jul 2025).

7. Significance in Remote Sensing Model Design

SkySense V2 demonstrates that a single transformer backbone, judiciously augmented with targeted architectural and self-supervision innovations, can yield state-of-the-art performance across heterogeneous remote sensing modalities and tasks. Its contributions include: demonstrating the value of large-scale, modality-agnostic pre-training; articulating the effectiveness of prompt-driven modality conditioning; and providing a scalable paradigm for parameter-efficient multi-modal fusion. These results establish a template for further foundation model development in Earth observation domains, with prospective impact analogous to that of vision-language pre-training in the natural image domain (Zhang et al., 18 Jul 2025).