---
title: 'SkyCaptioner-V1: Remote Sensing Captioning'
url: https://www.emergentmind.com/topics/skycaptioner-v1
type: topic
---

# SkyCaptioner-V1: Remote Sensing Captioning

SkyCaptioner-V1 is a pipeline for enhancing remote sensing image caption datasets and training captioning models, with a focus on leveraging large language models (LLMs) to augment and correct human-written captions. Designed around the Remote Sensing Image Captioning Dataset (RSICD), SkyCaptioner-V1 addresses core limitations in dataset diversity, grammaticality, and representational richness, using an LLM-based preprocessing stage to yield improved downstream captioning performance [2312.10905].

## 1. Dataset Specification and Preprocessing

SkyCaptioner-V1 centers its methodology on the RSICD, comprising approximately 10,000 satellite images at $224 \times 224$ resolution, each annotated with five human-written captions. The dataset spans 30 semantic scene classes, such as airport, beach, bridge, park, and stadium. Crucially, over 60% of captions are exact duplicates, and the original vocabulary is moderate in size (2,643 unique tokens). 

Preprocessing entails canonical lowercasing, punctuation removal (except for "<start>" and "<end>" markers), whitespace tokenization, dictionary construction (with a frequency threshold of 5; infrequent words are mapped to "<unk>"), sequence padding/truncation to length $T$ (commonly $T=20$), and the insertion of start/end markers. Data are split at the image level, typically using an 80/20 train/validation partition.

## 2. LLM-Based Caption Augmentation

A foundational component of SkyCaptioner-V1 is the incorporation of LLM-based caption postprocessing, using OpenAI's “gpt-3.5-turbo” via the ChatGPT API with temperature set to 1.0. The process follows a structured prompting protocol:

- **System Prompt:** "You are a helpful assistant that follows instructions extremely well."
- **User Prompt Template:** "The following sentence may have several grammatical errors; please respond with a grammatically correct sentence that means the same thing and, if possible, is more concise."
- **Input:** Original RSICD caption
- **Output:** Corrected/paraphrased caption (e.g., "many planes are parked next to a long building in an airport ." becomes "Several planes are parked alongside a lengthy building at the airport.")

No LLM fine-tuning is performed. The use of LLM augmentation produces marked changes in dataset properties:

| Metric                    | Original | Augmented |
|---------------------------|----------|-----------|
| Vocabulary size           | 2,643    | 5,365     |
| Misspelling ratio (%)     | 1.04     | 0.70      |
| One-time tokens           | 925      | 2,102     |

Further, n-gram repetition is reduced, and paraphrasing enhances descriptive diversity.

## 3. Model Architecture and Training Protocol

SkyCaptioner-V1 employs a standard encoder-decoder architecture with visual attention, following the “Show, Attend and Tell” paradigm:

- **Encoder:** Transfer-learned CNN distilled to $14 \times 14$ spatial features. ResNet-101 (2048 channels), VGG-16/19 (512 channels), and MnasNet (320 channels for mobile deployment) are supported.
- **Decoder:** LSTM equipped with an attention MLP over encoder feature maps. Word embeddings and LSTM hidden states have dimension 512. Decoder output is a vocabulary-level softmax, with $V \approx 6,000$.
- **Training Details:** Cross-entropy loss at each step,
  $$
  L_{CE} = -\sum_{t=1}^T \log p(w_t^* | w_{<t}^*, I)
  $$
  Adam optimizer ($\beta_1=0.9, \beta_2=0.999$), learning rate $4 \times 10^{-4}$ with decay (0.8 every 5 epochs), batch size 64, regularization via dropout (0.5), and early stopping on validation METEOR after 30–50 epochs.

## 4. Evaluation Methodology and Results

Evaluation is conducted on standard text generation metrics, defined as follows:

- **BLEU-N:** Measures n-gram precision with a brevity penalty:
  $$
  p_n = \frac{\sum_{\text{n-gram} \in H} \min(\text{count}_H(\text{n-gram}), \max_{r\in R} \text{count}_r(\text{n-gram}))}{\sum_{\text{n-gram} \in H} \text{count}_H(\text{n-gram})}
  $$
- **METEOR:** Favors recall and synonymy (Banerjee & Lavie, 2005), with
  $$
  P = \frac{|\text{matched tokens}|}{|H|}, \quad R = \frac{|\text{matched tokens}|}{|R|} \\
  F_{mean} = \frac{(1+\alpha)PR}{R + \alpha P},\ \alpha=9
  $$
- **ROUGE-L:** Based on longest common subsequence,
  $$
  P_{LCS} = \frac{LCS(H, R)}{|H|}, \quad R_{LCS} = \frac{LCS(H, R)}{|R|} \\
  ROUGE\_L = \frac{(1+\beta^2)P_{LCS}R_{LCS}}{R_{LCS} + \beta^2 P_{LCS}},\ \beta=1.2
  $$

Quantitative results demonstrate consistent improvement with LLM-augmented captions. For instance, ResNet-101 achieves METEOR 0.7033 (augmented) versus 0.6859 (original), BLEU-1 0.528 (augmented) versus 0.512 (original), and ROUGE-L 0.638 (augmented) versus 0.622 (original). MnasNet exhibits the largest METEOR gain (+0.0534), indicating particular downstream benefit to smaller models.

## 5. Analysis, Limitations, and Best Practices

LLM-based grammar correction and paraphrasing reliably increases caption diversity, reduces overfitting to common n-grams, and improves metric performance, especially for models with lower capacity such as MnasNet. METEOR is identified as a preferable primary metric due to reduced sensitivity to increased reference diversity, compared to BLEU.

Nevertheless, the augmentation procedure increases the incidence of rare, one-time tokens (from 925 to 2,102), which can negatively affect model robustness for tail vocabulary. Prompt configuration is limited to default ChatGPT settings, and experiments are constrained to English-language captions in RSICD.

Recommended practices include temperature sweeps (0.5–1.5) to control paraphrastic creativity; post-augmentation filtering of rare tokens or use of subword tokenization (e.g., Byte-Pair Encoding); extension to transformer-based encoders; evaluation with larger attention decoders; pursuit of multilingual captioning via LLMs; and potential integration of reinforcement learning (e.g., CIDEr optimization) to maximize downstream caption metrics.

## 6. Extensions and Future Directions

A plausible implication is that LLM augmentation can generalize to other domains characterized by low-diversity or noisy captions. Evaluation with vision transformer architectures and cross-linguistic generalization via multilingual LLMs remain open research directions. Direct metric-driven training via reinforcement learning presents a potential avenue for further boosting performance beyond supervised cross-entropy objectives [2312.10905].

By following the SkyCaptioner-V1 pipeline, encompassing LLM-based data enhancement, careful preprocessing, and rigorous model evaluation, it is possible to advance the quality and applicability of image captioning systems for specialized domains such as remote sensing.

Source: https://www.emergentmind.com/topics/skycaptioner-v1