SkySense: Multi-Modal Remote Sensing Model

Updated 8 March 2026

SkySense is a family of multi-modal remote sensing models that integrate optical, multispectral, and SAR data for comprehensive Earth observation analysis.
Its architecture features modality-specific backbones, transformer fusion with geo-context attention, and adaptive patch merging to handle diverse spatial-temporal scales.
Pre-training employs contrastive and alignment strategies on 21.5 million spatiotemporal sequences, achieving superior performance across 16 remote sensing benchmarks.

SkySense denotes a family of large-scale multi-modal foundation models for remote sensing (RS), explicitly engineered for universal interpretation of Earth observation imagery across diverse modalities, temporal scales, and tasks. The SkySense lineage establishes new standards in the synthesis of optical, multispectral, and synthetic aperture radar (SAR) data, incorporates geo-contextual and temporal information, and advances foundation model design for remote-sensing applications from pixel-level segmentation to scene-level multi-modal analysis (Guo et al., 2023, Zhang et al., 18 Jul 2025, Luo et al., 2024).

The original SkySense (V1) model architecture is based on a factorized multi-modal spatiotemporal encoder. Spatial feature extraction is realized via three modality-specific backbones: Swin-Huge for high-resolution optical (RGB) tiles, ViT-Large for Sentinel-2 multispectral sequences, and ViT-Large for Sentinel-1 SAR sequences. These encoders map each modality $x_i$ into feature tensors

$F_i = g_i(x_i) \in \mathbb{R}^{h \times w \times T_i \times d},\quad i\in\{\text{HR},\text{MS},\text{SAR}\},$

where $h\times w$ is the spatial resolution post-patching, $T_i$ is the number of time-steps, and $d$ the feature dimension.

All modalities are concatenated into a single sequence after flattening spatial dimensions,

$F_T = \mathrm{Concat}[F_\text{HR}, F_\text{MS}, F_\text{SAR}] \in \mathbb{R}^{N_S \times N_T \times d}.$

A learnable date-aware positional encoding is applied, followed by a multi-layer transformer that fuses spatial, temporal, and cross-modal dependencies. An optional geo-context attention mechanism introduces region-specific prototypes, resulting in feature vectors of dimension $2d$ per spatial token (Guo et al., 2023).

SkySense V2 transitions to a unified transformer backbone across modalities, leveraging a staged hierarchy (SwinV2 in early stages, ViT in later) and explicit modality prompt tokens (“MsP”) for enhanced feature diversity. An adaptive patch merging (APM) module harmonizes spatial scales from disparate sensor resolutions, and a Mixture-of-Experts (MoE) layer in the upper transformer stages improves parameter efficiency and modeling capacity (Zhang et al., 18 Jul 2025).

2. Pre-Training Methodologies

SkySense employs large-scale self-supervised pre-training on curated datasets. The first generation (V1) ingests 21.5 million spatiotemporal sequences combining WorldView-3/4 static RGB (GSD ≈ 0.3 m), Sentinel-2 multispectral time-series (10 m, ≈20 frames), and Sentinel-1 SAR time-series (10 m, ≈10 frames), encompassing 8.8 million km² over 40+ countries (Guo et al., 2023).

Three main pre-training strategies underpin SkySense V2:

Multi-Granularity Contrastive Learning (MGCL): Enforces consistency between pixel, object (via clustering and Sinkhorn-Knopp), and image-level features across two augmented views. The loss aggregates over modalities and the multi-modal fused token.
Dense Image–Text Alignment (ITA): Aligns pixel-level features to OpenStreetMap text labels encoded by a CLIP text encoder, facilitating semantic transfer to tasks where textual context is present.
Query-based Semantic Aggregation Contrastive Learning (QSACL): Employs learnable queries that aggregate local and global features from image crops and full views, optimized by a cross-view contrastive objective (Zhang et al., 18 Jul 2025).

SkySense uses a teacher–student EMA paradigm for stable pre-training.

3. Geo-Context and Temporal Modeling

Geo-context integration is achieved via Geo-Context Prototype Learning (GCPL). The globe is partitioned into $R=4096$ regions, each holding $N_p=100$ learnable region-specific prototypes $P_r \in \mathbb{R}^{N_p \times d}$ . During pre-training, each sample’s fused feature tokens are matched to prototypes via cosine similarity, balanced assignment (Sinkhorn-Knopp), and exponential moving average (EMA) updates. This mechanism imparts region-awareness that supports downstream tasks requiring sensitivity to geographic semantics (Guo et al., 2023).

Temporal information is treated explicitly: time-series from Sentinel-2 and Sentinel-1 are processed as ordered sequences; a learnable date table provides acquisition-aware positional embeddings. This enables downstream modeling of seasonal patterns, phenology, or incremental change (Guo et al., 2023, Zhang et al., 18 Jul 2025).

4. Benchmarks, Empirical Results, and Modularity

SkySense and its successors are evaluated on 16 datasets spanning 7 tasks: segmentation (mean IoU), object detection (mAP@50), change detection (mean F₁), classification (OA/mAP), multi-modal segmentation, multi-modal classification, and scene graph-related tasks. SkySense V1 consistently outperforms 18 competitive RSFMs, including GFM, SatLas, and Scale-MAE: e.g., +2.76 pp over GFM, +3.67 pp over SatLas, +3.61 pp over Scale-MAE on average (Guo et al., 2023). V2 yields a further mean gain of 1.8 points on these benchmarks (Zhang et al., 18 Jul 2025).

Ablation studies reveal:

Each architecture module yields distinct advantages (e.g., MGCL +2.2 mIoU, multi-modal +2.6 mIoU, geo-context +0.5 mIoU on Dyna-MM).
Even for single-modality tasks, multi-modal pre-training provides measurable improvements (+1.9 pp on iSAID segmentation).

SkySense design is modular: encoders can be composed for single/multi-modal, static/temporal, geo-aware or generic use-cases.

5. Vision-Language Extension: SkySenseGPT and FIT-RS

SkySenseGPT, a derivative modality in the SkySense family, targets vision-language tasks using instruction tuning on the FIT-RS dataset (1.8M samples across 11 fine-grained tasks). Its architecture fuses a frozen CLIP-ViT-L/14 vision encoder, a multimodal projector, and Vicuna-v1.5 LLM fine-tuned via LoRA adapters. SkySenseGPT is benchmarked on FIT-RSFG/FIT-RSRC and public datasets (AID, SIRI-WHU, RSVQA-HR/LR), surpassing prior RSLMMs in both basic tasks (captioning, VQA, classification) and relation reasoning (SGG recall: 17% region, 9.6% image; relation detection F1: 88.7%) (Luo et al., 2024).

Instruction data ranges from object/region captioning, VQA, and classification, to relation detection/reasoning, object reasoning, and scene-graph generation. Exposing the model to graduated relation-difficulty tasks fosters robust internal modeling of multi-object spatial/logical relations (Luo et al., 2024).

6. Comparative Table: Selected SkySense Benchmarks

Task	Prev. SOTA (%)	SkySense V1 (%)	SkySense V2 (%)	Gain (V2–V1)
Segmentation (avg mIoU)	91.16	92.76	93.6*	+0.84*
Detection (avg mAP₅₀)	72.45	77.19	78.1*	+0.91*
Change Detection (F₁)	89.32	91.21	93.1*	+1.89*
Cls (avg OA)	94.63	96.11	98.34**	+2.23**

*Estimates from per-dataset breakdowns in (Zhang et al., 18 Jul 2025). **AID dataset, 20% split.

7. Open Challenges and Future Research

SkySense identifies several avenues for further exploration:

Incorporating natural-language information (e.g., crowd-labeled captions, geospatial reports) into foundation modeling.
Scaling to longer time-series or higher spatial resolutions; domain adaptation to non-optical modalities (hyperspectral, LiDAR).
Integrating physical models (e.g., atmospheric or radiative transfer models) into representation learning.
In vision-LLMs, extending relation vocabularies beyond fixed subcategories and addressing visual grounding/hallucination risks (Guo et al., 2023, Luo et al., 2024).

A plausible implication is that, as Earth observation data volume and diversity accelerate, unified multi-modal and language-integrated foundation models will become foundational infrastructure for automated, semantics-rich remote-sensing interpretation.

References:

(Guo et al., 2023) "SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery"
(Zhang et al., 18 Jul 2025) "SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensing"
(Luo et al., 2024) "SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding"