SkyCaptioner-V1: Remote Sensing Captioning

• SkyCaptioner-V1 is a pipeline that enhances remote sensing caption datasets by using LLM-based preprocessing to correct grammar and enrich descriptions.
• It meticulously preprocesses the RSICD dataset, increasing vocabulary size and reducing n-gram repetition to improve caption diversity.
• The system employs an encoder-decoder model with visual attention, achieving measurable gains on BLEU, METEOR, and ROUGE metrics, particularly for lightweight architectures.

SkyCaptioner-V1 is a pipeline for enhancing remote sensing image caption datasets and training captioning models, with a focus on leveraging LLMs to augment and correct human-written captions. Designed around the Remote Sensing Image Captioning Dataset (RSICD), SkyCaptioner-V1 addresses core limitations in dataset diversity, grammaticality, and representational richness, using an LLM-based preprocessing stage to yield improved downstream captioning performance (Rosario et al., 2023).

1. Dataset Specification and Preprocessing

SkyCaptioner-V1 centers its methodology on the RSICD, comprising approximately 10,000 satellite images at $224 \times 224$ resolution, each annotated with five human-written captions. The dataset spans 30 semantic scene classes, such as airport, beach, bridge, park, and stadium. Crucially, over 60% of captions are exact duplicates, and the original vocabulary is moderate in size (2,643 unique tokens).

Preprocessing entails canonical lowercasing, punctuation removal (except for "<start>" and "<end>" markers), whitespace tokenization, dictionary construction (with a frequency threshold of 5; infrequent words are mapped to "<unk>"), sequence padding/truncation to length $T$ (commonly $T=20$), and the insertion of start/end markers. Data are split at the image level, typically using an 80/20 train/validation partition.

2. LLM-Based Caption Augmentation

A foundational component of SkyCaptioner-V1 is the incorporation of LLM-based caption postprocessing, using OpenAI's “gpt-3.5-turbo” via the ChatGPT API with temperature set to 1.0. The process follows a structured prompting protocol:

• System Prompt: "You are a helpful assistant that follows instructions extremely well."
• User Prompt Template: "The following sentence may have several grammatical errors; please respond with a grammatically correct sentence that means the same thing and, if possible, is more concise."
• Input: Original RSICD caption
• Output: Corrected/paraphrased caption (e.g., "many planes are parked next to a long building in an airport ." becomes "Several planes are parked alongside a lengthy building at the airport.")

No LLM fine-tuning is performed. The use of LLM augmentation produces marked changes in dataset properties:

Metric	Original	Augmented
Vocabulary size	2,643	5,365
Misspelling ratio (%)	1.04	0.70
One-time tokens	925	2,102

Further, n-gram repetition is reduced, and paraphrasing enhances descriptive diversity.

3. Model Architecture and Training Protocol

SkyCaptioner-V1 employs a standard encoder-decoder architecture with visual attention, following the “Show, Attend and Tell” paradigm:

• Encoder: Transfer-learned CNN distilled to $14 \times 14$ spatial features. ResNet-101 (2048 channels), VGG-16/19 (512 channels), and MnasNet (320 channels for mobile deployment) are supported.
• Decoder: LSTM equipped with an attention MLP over encoder feature maps. Word embeddings and LSTM hidden states have dimension 512. Decoder output is a vocabulary-level softmax, with $V \approx 6,000$.
• Training Details: Cross-entropy loss at each step,

$L_{CE} = -\sum_{t=1}^T \log p(w_t^* | w_{<t}^*, I)$

Adam optimizer ($\beta_1=0.9, \beta_2=0.999$), learning rate $4 \times 10^{-4}$ with decay (0.8 every 5 epochs), batch size 64, regularization via dropout (0.5), and early stopping on validation METEOR after 30–50 epochs.

4. Evaluation Methodology and Results

Evaluation is conducted on standard text generation metrics, defined as follows:

• BLEU-N: Measures n-gram precision with a brevity penalty:

$$p_n = \frac{\sum_{\text{n-gram} \in H} \min(\text{count}_H(\text{n-gram}), \max_{r\in R} \text{count}_r(\text{n-gram}))}{\sum_{\text{n-gram} \in H} \text{count}_H(\text{n-gram})}$$

• METEOR: Favors recall and synonymy (Banerjee & Lavie, 2005), with

$$P = \frac{|\text{matched tokens}|}{|H|}, \quad R = \frac{|\text{matched tokens}|}{|R|} \ F_{mean} = \frac{(1+\alpha)PR}{R + \alpha P},\ \alpha=9$$

• ROUGE-L: Based on longest common subsequence,

$T$0

Quantitative results demonstrate consistent improvement with LLM-augmented captions. For instance, ResNet-101 achieves METEOR 0.7033 (augmented) versus 0.6859 (original), BLEU-1 0.528 (augmented) versus 0.512 (original), and ROUGE-L 0.638 (augmented) versus 0.622 (original). MnasNet exhibits the largest METEOR gain (+0.0534), indicating particular downstream benefit to smaller models.

5. Analysis, Limitations, and Best Practices

LLM-based grammar correction and paraphrasing reliably increases caption diversity, reduces overfitting to common n-grams, and improves metric performance, especially for models with lower capacity such as MnasNet. METEOR is identified as a preferable primary metric due to reduced sensitivity to increased reference diversity, compared to BLEU.

Nevertheless, the augmentation procedure increases the incidence of rare, one-time tokens (from 925 to 2,102), which can negatively affect model robustness for tail vocabulary. Prompt configuration is limited to default ChatGPT settings, and experiments are constrained to English-language captions in RSICD.

Recommended practices include temperature sweeps (0.5–1.5) to control paraphrastic creativity; post-augmentation filtering of rare tokens or use of subword tokenization (e.g., Byte-Pair Encoding); extension to transformer-based encoders; evaluation with larger attention decoders; pursuit of multilingual captioning via LLMs; and potential integration of reinforcement learning (e.g., CIDEr optimization) to maximize downstream caption metrics.

6. Extensions and Future Directions

A plausible implication is that LLM augmentation can generalize to other domains characterized by low-diversity or noisy captions. Evaluation with vision transformer architectures and cross-linguistic generalization via multilingual LLMs remain open research directions. Direct metric-driven training via reinforcement learning presents a potential avenue for further boosting performance beyond supervised cross-entropy objectives (Rosario et al., 2023).

By following the SkyCaptioner-V1 pipeline, encompassing LLM-based data enhancement, careful preprocessing, and rigorous model evaluation, it is possible to advance the quality and applicability of image captioning systems for specialized domains such as remote sensing.