RS-CoCa: Remote Sensing Captioning Model

• RS-CoCa is a family of vision–language foundation models that specializes in generating detailed captions for remote sensing images using a two-stream (vision encoder and text decoder) architecture.
• It leverages full fine-tuning of a ViT-L-14 based vision encoder and an autoregressive transformer decoder on the HQRS-IT-210K dataset to enhance semantic alignment.
• The model achieves superior caption quality with high METEOR, ROUGE-L, and SPICE scores by optimizing a dual loss function that balances contrastive and captioning objectives.

RS-CoCa designates a family of vision–language foundation models adapted from the Contrastive Captioner (CoCa) architecture for remote sensing applications, with a specific emphasis on high-quality remote-sensing image captioning. These models are fine-tuned on large, high-quality image–text datasets constructed using advanced data curation and summarization techniques, enabling superior semantic alignment and descriptive quality compared to baseline captioning approaches in the remote sensing domain (He et al., 22 Jul 2025).

1. Model Architecture and Instantiation

RS-CoCa retains the canonical CoCa two-stream design, integrating:

• Vision Encoder: A Vision Transformer (ViT-L-14) backbone that generates a fixed-length visual feature vector $z_v$ per image.
• Text Decoder: An autoregressive transformer decoder that, conditioned on $z_v$ and prior tokens, generates the caption autoregressively.
• Contrastive Head: Independent projection layers for image and text producing $z_v$ and $z_t$, with $\ell_2$ normalization for contrastive alignment.
• Generation Head: The same decoder drives the captioning loss.

No architectural modifications are introduced relative to CoCa; however, all weights—including those of the vision encoder, decoder, and projection heads—are fully fine-tuned on the remote-sensing dataset HQRS-IT-210K (He et al., 22 Jul 2025).

Component	Architecture	Fine-tuning Strategy
Vision Encoder	ViT-L-14 (CLIP style)	Full fine-tuning
Text Decoder	Transformer, autoregressive	Full fine-tuning
Contrastive/Generation Heads	Standard CoCa	Full fine-tuning

Key distinction: RS-CoCa is domain-adapted via fine-tuning exclusively on high-quality, multi-perspective remote-sensing image–text pairs, diverging from the original CoCa's natural image training paradigm.

2. Training Objectives and Loss Functions

RS-CoCa optimizes a dual-objective function, directly inherited from CoCa:

• Contrastive Loss (InfoNCE-based):

$$\mathcal{L}_{\text{con}} = -\frac{1}{N} \sum_{i=1}^N \left[ \log \frac{\exp(z_{iv}^\top z_{it} / \tau)}{\sum_{j=1}^N \exp(z_{iv}^\top z_{jt} / \tau)} + \log \frac{\exp(z_{it}^\top z_{iv} / \tau)}{\sum_{j=1}^N \exp(z_{it}^\top z_{jv} / \tau)} \right]$$

where $z_{iv}$, $z_{it}$ are the normalized image/text embeddings, $N$ is batch size, and $\tau$ is the temperature.

• Captioning Loss (Tokenwise Cross-Entropy):

$z_v$0

where $z_v$1 is the input image and $z_v$2 ground-truth token at position $z_v$3.

• Loss Weights: During HQRS-IT-210K pretraining, the composite objective is

$z_v$4

with $z_v$5, $z_v$6. Caption objectives are upweighted in downstream captioning tasks ($z_v$7, $z_v$8) (He et al., 22 Jul 2025).

3. Dataset Construction and Fine-Tuning Regimen

RS-CoCa is trained on HQRS-IT-210K, a comprehensive synthetic dataset comprising:

• Images: 210,556 remote-sensing images sourced from 23 public satellite and UAV datasets, spanning classification, detection, and segmentation tasks.
• Captions: Approximately 1.26 million high-diversity captions (~6 per image), each generated by multi-perspective MLLM/LLM relays and integrated into longer, richer descriptions (mean length 35.6 words).

Fine-tuning uses AdamW (learning rate $z_v$9, batch size 32, mixed-precision, RTX 4090 GPU), with all model parameters updated. No data augmentation or explicit image preprocessing is reported beyond standard normalization (He et al., 22 Jul 2025).

4. Quantitative Performance on Remote-Sensing Benchmarks

RS-CoCa attains state-of-the-art captioning results on standard benchmarks:

Dataset	Method	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
RSICD	BITA	0.504	0.420	0.717	3.045	0.548
	RS-CapRet	0.455	0.376	0.649	2.605	0.484
	RS-CoCa	0.449	0.425	0.708	2.517	0.555
UCM	BITA	0.719	0.469	0.838	3.845	0.549
	RS-CapRet	0.645	0.447	0.786	3.429	0.525
	RS-CoCa	0.734	0.487	0.842	3.701	0.539

RS-CoCa achieves:

• The highest METEOR, ROUGE-L, and SPICE scores on both RSICD and UCM-Caption datasets.
• Comparable BLEU scores to the best baselines.
• Robust CIDEr scores, supporting vocabulary diversity and semantic correspondence (He et al., 22 Jul 2025).

5. Qualitative Evaluation and Descriptive Capacity

RS-CoCa’s generative captions are significantly richer and semantically denser than manual ground-truths or baseline models:

• Urban Residential Example:
  • GT: "Residential area with rectangular buildings and roads."
  • RS-CoCa: "A dense cluster of multistory rectangular apartment buildings arranged in a grid pattern, interconnected by narrow asphalt roads, with sporadic patches of landscaped greenery and parked vehicles lining the sidewalks."
• Agricultural Example:
  • GT: "Fields with irrigation canal."
  • RS-CoCa: "An aerial view of elongated agricultural plots in varying shades of green, bisected by a sinuous irrigation canal running diagonally from the bottom left to the top right, with adjacent roads and scattered farm structures visible on the eastern edge."

The model generates fine-grained details pertaining to spatial relations, object types, geometry, and scene context, often exceeding the level of detail present in human annotations (He et al., 22 Jul 2025).

6. Ablation Insights and Model Optimizations

Key findings relevant to RS-CoCa’s efficacy:

• Caption Length and Richness: Longer, detailed captions (~50 words) enhance the semantical alignment between generated text and image content when measured by CLIP similarity distributions.
• Prompt Compression: Summarization prompts that fuse multiple raw descriptions into a concise, single sentence (≤77 tokens) maintain rich coverage while respecting tokenization constraints critical for both the contrastive and generative branches.
• Objective Weighting: Emphasizing the generative objective (caption loss) during task-specific fine-tuning yields marked improvements in semantic correspondence and descriptive quality.

A plausible implication is that richer, more comprehensive captions during pretraining systematically bootstrap downstream generative capacity.

7. Limitations and Prospective Extensions

Identified limitations include:

• Hallucination: Despite prompt engineering and dataset filtering, there remains some tendency for the generative model to introduce non-observable or erroneous details (hallucinations). Potential remedies include retrieval-augmented grounding or further alignment-based regularization.
• Domain Coverage: RS-CoCa is currently focused on satellite/UAV RGB imagery within HQRS-IT-210K. Expansion to multispectral, SAR, or time-series modalities is noted as a future direction.
• Model Scaling and Tasks: Current experiments utilize the CoCa ViT-L-14 backbone. Larger vision–LLMs (e.g., Flamingo, GPT-4V) and new annotation regimes (region grounding, question answering, instruction tuning) may further extend applicability (He et al., 22 Jul 2025).

In summary, RS-CoCa demonstrates the critical role of high-quality, diverse remote sensing image–text pairs and tailored generative objective weighting in achieving high-fidelity, semantically aligned image captions for challenging aerial and satellite imagery scenarios.