---
title: 'RS-CoCa: Remote Sensing Captioning Model'
url: https://www.emergentmind.com/topics/rs-coca
type: topic
---

# RS-CoCa: Remote Sensing Captioning Model

RS-CoCa designates a family of vision–language foundation models adapted from the Contrastive Captioner (CoCa) architecture for remote sensing applications, with a specific emphasis on high-quality remote-sensing image captioning. These models are fine-tuned on large, high-quality image–text datasets constructed using advanced data curation and summarization techniques, enabling superior semantic alignment and descriptive quality compared to baseline captioning approaches in the remote sensing domain [2507.16716].

## 1. Model Architecture and Instantiation

RS-CoCa retains the canonical CoCa two-stream design, integrating:

- **Vision Encoder:** A Vision Transformer (ViT-L-14) backbone that generates a fixed-length visual feature vector $z_v$ per image.
- **Text Decoder:** An autoregressive transformer decoder that, conditioned on $z_v$ and prior tokens, generates the caption autoregressively.
- **Contrastive Head:** Independent projection layers for image and text producing $z_v$ and $z_t$, with $\ell_2$ normalization for contrastive alignment.
- **Generation Head:** The same decoder drives the captioning loss.

No architectural modifications are introduced relative to CoCa; however, all weights—including those of the vision encoder, decoder, and projection heads—are fully fine-tuned on the remote-sensing dataset HQRS-IT-210K [2507.16716].

| Component               | Architecture            | Fine-tuning Strategy          |
|-------------------------|------------------------|------------------------------|
| Vision Encoder          | ViT-L-14 (CLIP style)  | Full fine-tuning             |
| Text Decoder            | Transformer, autoregressive | Full fine-tuning       |
| Contrastive/Generation Heads | Standard CoCa      | Full fine-tuning             |

Key distinction: RS-CoCa is domain-adapted via fine-tuning exclusively on high-quality, multi-perspective remote-sensing image–text pairs, diverging from the original CoCa's natural image training paradigm.

## 2. Training Objectives and Loss Functions

RS-CoCa optimizes a dual-objective function, directly inherited from CoCa:

- **Contrastive Loss (InfoNCE-based):**
  \[
  \mathcal{L}_{\text{con}} = -\frac{1}{N} \sum_{i=1}^N \left[ 
    \log \frac{\exp(z_{iv}^\top z_{it} / \tau)}{\sum_{j=1}^N \exp(z_{iv}^\top z_{jt} / \tau)} + 
    \log \frac{\exp(z_{it}^\top z_{iv} / \tau)}{\sum_{j=1}^N \exp(z_{it}^\top z_{jv} / \tau)}
  \right]
  \]
  where $z_{iv}$, $z_{it}$ are the normalized image/text embeddings, $N$ is batch size, and $\tau$ is the temperature.
- **Captioning Loss (Tokenwise Cross-Entropy):**
  \[
  \mathcal{L}_{\text{cap}} = -\sum_{t=1}^T \log\, p(x_t | I, x_{<t}),
  \]
  where $I$ is the input image and $x_t$ ground-truth token at position $t$.
- **Loss Weights:** During HQRS-IT-210K pretraining, the composite objective is
  \[
  \mathcal{L}_{\text{total}} = \lambda_{\text{con}}\,\mathcal{L}_{\text{con}} + \lambda_{\text{cap}}\,\mathcal{L}_{\text{cap}},
  \]
  with $\lambda_{\text{con}}=1$, $\lambda_{\text{cap}}=2$. Caption objectives are upweighted in downstream captioning tasks ($\lambda_{\text{con}}=0$, $\lambda_{\text{cap}}=1$) [2507.16716].

## 3. Dataset Construction and Fine-Tuning Regimen

RS-CoCa is trained on HQRS-IT-210K, a comprehensive synthetic dataset comprising:

- **Images:** 210,556 remote-sensing images sourced from 23 public satellite and UAV datasets, spanning classification, detection, and segmentation tasks.
- **Captions:** Approximately 1.26 million high-diversity captions (~6 per image), each generated by multi-perspective MLLM/LLM relays and integrated into longer, richer descriptions (mean length 35.6 words).

Fine-tuning uses AdamW (learning rate $2\times10^{-5}$, batch size 32, mixed-precision, RTX 4090 GPU), with all model parameters updated. No data augmentation or explicit image preprocessing is reported beyond standard normalization [2507.16716].

## 4. Quantitative Performance on Remote-Sensing Benchmarks

RS-CoCa attains state-of-the-art captioning results on standard benchmarks:

| Dataset  | Method      | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
|----------|------------|--------|--------|---------|-------|-------|
| RSICD    | BITA       | 0.504  | 0.420  | 0.717   | 3.045 | 0.548 |
|          | RS-CapRet  | 0.455  | 0.376  | 0.649   | 2.605 | 0.484 |
|          | **RS-CoCa**| 0.449  | 0.425  | 0.708   | 2.517 | 0.555 |
| UCM      | BITA       | 0.719  | 0.469  | 0.838   | 3.845 | 0.549 |
|          | RS-CapRet  | 0.645  | 0.447  | 0.786   | 3.429 | 0.525 |
|          | **RS-CoCa**| 0.734  | 0.487  | 0.842   | 3.701 | 0.539 |

RS-CoCa achieves:

- The highest METEOR, ROUGE-L, and SPICE scores on both RSICD and UCM-Caption datasets.
- Comparable BLEU scores to the best baselines.
- Robust CIDEr scores, supporting vocabulary diversity and semantic correspondence [2507.16716].

## 5. Qualitative Evaluation and Descriptive Capacity

RS-CoCa’s generative captions are significantly richer and semantically denser than manual ground-truths or baseline models:

- **Urban Residential Example:**
  - **GT:** "Residential area with rectangular buildings and roads."
  - **RS-CoCa:** "A dense cluster of multistory rectangular apartment buildings arranged in a grid pattern, interconnected by narrow asphalt roads, with sporadic patches of landscaped greenery and parked vehicles lining the sidewalks."
- **Agricultural Example:**
  - **GT:** "Fields with irrigation canal."
  - **RS-CoCa:** "An aerial view of elongated agricultural plots in varying shades of green, bisected by a sinuous irrigation canal running diagonally from the bottom left to the top right, with adjacent roads and scattered farm structures visible on the eastern edge."

The model generates fine-grained details pertaining to spatial relations, object types, geometry, and scene context, often exceeding the level of detail present in human annotations [2507.16716].

## 6. Ablation Insights and Model Optimizations

Key findings relevant to RS-CoCa’s efficacy:

- **Caption Length and Richness:** Longer, detailed captions (~50 words) enhance the semantical alignment between generated text and image content when measured by CLIP similarity distributions.
- **Prompt Compression:** Summarization prompts that fuse multiple raw descriptions into a concise, single sentence (≤77 tokens) maintain rich coverage while respecting tokenization constraints critical for both the contrastive and generative branches.
- **Objective Weighting:** Emphasizing the generative objective (caption loss) during task-specific fine-tuning yields marked improvements in semantic correspondence and descriptive quality.

A plausible implication is that richer, more comprehensive captions during pretraining systematically bootstrap downstream generative capacity.

## 7. Limitations and Prospective Extensions

Identified limitations include:

- **Hallucination:** Despite prompt engineering and dataset filtering, there remains some tendency for the generative model to introduce non-observable or erroneous details (hallucinations). Potential remedies include retrieval-augmented grounding or further alignment-based regularization.
- **Domain Coverage:** RS-CoCa is currently focused on satellite/UAV RGB imagery within HQRS-IT-210K. Expansion to multispectral, SAR, or time-series modalities is noted as a future direction.
- **Model Scaling and Tasks:** Current experiments utilize the CoCa ViT-L-14 backbone. Larger vision–language models (e.g., Flamingo, GPT-4V) and new annotation regimes (region grounding, question answering, instruction tuning) may further extend applicability [2507.16716].

In summary, RS-CoCa demonstrates the critical role of high-quality, diverse remote sensing image–text pairs and tailored generative objective weighting in achieving high-fidelity, semantically aligned image captions for challenging aerial and satellite imagery scenarios.

Source: https://www.emergentmind.com/topics/rs-coca