RSTeller: Large-Scale Remote Sensing Captions

• RSTeller is a large-scale, multimodal dataset and pipeline that fuses aerial imagery from GEE with geospatial tags from OSM to generate descriptive captions using LLMs.
• It automates the creation of image–caption pairs, significantly reducing manual annotation costs and enabling scalable remote sensing visual–language modeling.
• Experimental results demonstrate improved zero-shot classification and retrieval performance, highlighting the dataset’s efficacy in various remote sensing tasks.

RSTeller is a large-scale, multimodal dataset and data generation pipeline for remote sensing (RS) image–caption pairs, designed to address the scarcity of richly annotated visual–language resources in the RS domain. Built by augmenting openly available RS imagery from Google Earth Engine (GEE) with geospatial semantic data from OpenStreetMap (OSM), RSTeller leverages LLMs to generate semantically rich, descriptive captions for each RS image patch. The resulting corpus, comprising over 1.19 million aerial images and more than 2.53 million captions, enables improved training of vision–LLMs (VLMs) for numerous RS interpretation tasks and facilitates scalable, environmentally sustainable data annotation without manual expert effort (Ge et al., 2024).

1. Motivation and Challenges in Remote Sensing Visual–Language Modeling

Remote sensing visual–language modeling has been constrained by a deficit of large, high-quality, semantically dense image–text datasets. By contrast, natural image VLMs such as CLIP are trained on hundreds of millions of image–caption pairs. In RS, the largest prior caption dataset had ≲5 million image–text pairs, with most alternatives below 200,000. The creation of such resources is particularly costly, requiring domain experts and considerable manual annotation effort. This high annotation cost impedes model scaling and introduces environmental sustainability concerns due to redundant human labor. Scaling multimodal RS data from open sources has become increasingly critical, as data and model scale are empirically linked to downstream accuracy (“scaling laws” per Kaplan et al. 2020, Riquelme et al. 2021). Freely available RS repositories such as OSM and GEE offer global coverage but lack sufficiently curated paired data for VLM research. RSTeller addresses these challenges by automating image–caption pair generation at scale, reducing human annotation requirements, and lowering the barriers to entry in RS-VLM research (Ge et al., 2024).

2. Data Sources, Processing Pipeline, and Formalization

RSTeller fuses GEE’s NAIP aerial imagery (RGB, 0.6 m GSD, 448×448 pixel patches) with OSM’s geospatial tags (“ways,” “relations,” etc.) as input for LLM-driven caption generation. The pipeline consists of four sequential modules:

1. Raw Data Acquisition: Tiles from GEE are partitioned into 448×448 patches, each mapped to OSM geo-elements overlapping its geocoordinates.
2. Raw Caption Generation: For each valid patch, a primary OSM element is selected and a task-specific prompt is synthesized for the LLM, which returns a raw descriptive caption.
3. Caption Augmentation: The initial caption for each patch is augmented by eliciting up to four additional revisions or variants via LLM augmentation prompts, resulting in 2–5 captions per image.
4. Dataset Compilation: Captions are filtered for quality; valid image–caption sets are distributed across WebDataset shards for scalable consumption.

The dataset is formalized as: $D = \{(x_i,\{c_{i,j}\}_{j=1}^{m_i})\}_{i=1}^N$ where $x_i$ is the $i$-th RGB patch, $m_i$ the number of captions (2–5), and $N \approx 1.2 \times 10^6$. Each caption $c$ is a function of LLM outputs conditioned on interpretations and attributes of a selected OSM element. No manual caption writing occurs after initial data fetching (Ge et al., 2024).

3. Dataset Structure and Linguistic Properties

RSTeller encompasses 1,197,190 image patches and 2,539,256 paired captions. Image sources include NAIP (GEE), each patch RGB and 448×448 pixels, spanning the continental United States (August 2021–November 2022). Caption statistics denote a minimum length of 5 tokens (NLTK tokenization), a maximum of 192 tokens, mean length of 54.2, and a median of 48 tokens. The lexical diversity, as quantified by the MTLD score, more than doubles that of any previously available RS caption dataset, reflecting high semantic breadth. Caption content includes inferential terms ("likely," "possibly"), metric descriptors ("meters," "orientation"), and topical variation mirroring OSM key-tag distributions ("highway," "natural," "landuse"), which are long-tailed. This suggests that the dataset supports more granular and inferential visual–language modeling for RS (Ge et al., 2024).

4. Model Pretraining Protocols and Training Objectives

Continual pretraining is performed using OpenCLIP implementations of ViT-B/32 and ViT-L/14, with initial weights from established contrastive learning checkpoints: WIT (0.4B pairs), LAION-400M, DataComp (1.4B), or LAION-2B. Training employs the InfoNCE contrastive loss:

$$\mathcal{L}_{\rm InfoNCE} = - \frac{1}{|B|} \sum_{(i, t) \in B} \log \frac{\exp(\langle v_i, t_i \rangle / \tau)}{\sum_{t' \in B} \exp(\langle v_i, t' \rangle / \tau)} - \frac{1}{|B|} \sum_{(i, t) \in B} \log \frac{\exp(\langle v_i, t_i \rangle / \tau)}{\sum_{v' \in B} \exp(\langle v', t_i \rangle / \tau)}$$

where $v_i$ and $t_i$ are normalized vision/language embeddings and $\tau$ is a trainable temperature. Each batch (size 8,192 in scale-law experiments; 1,024 in checkpoint-effectiveness experiments, 4×A800 GPUs) is composed of one-third samples from LAION-10M and two-thirds from RS domain (RSTeller data). Adam optimizer with linear warm-up and cosine decay is employed, with learning rates of $x_i$0 (scale-law) or $x_i$1 (checkpoint); 5,000 or 20,000 training steps are standard, respectively. Notably, no supervised fine-tuning occurs; evaluation is strictly zero-shot (Ge et al., 2024).

5. Experimental Results and Performance Scaling

Downstream evaluation focuses on zero-shot single-label classification across seven RS datasets — AID, EuroSAT, fMoW, Million-AID, PatternNet, NWPU-RESISC45, RSI-CB256 — and text-to-image retrieval benchmarks — UCM Captions, RSICD, RSITMD. Ablation studies on scaling laws show that increasing RS-specific data from 25% to 100% (≈5M pairs) yields nearly linear reductions in top-1 classification error: average slope $x_i$2, $x_i$3. Retrieval error also diminishes with more domain data. Checkpoint-effectiveness analysis indicates that continual pretraining on RSTeller leads to consistent boosts: for ViT-L/14 (WIT-initialized), EuroSAT classification accuracy increases from 41.70% to 56.30% (+14.59), PatternNet from 71.32% to 77.14% (+5.82), and RSICD retrieval from 13.81 to 17.42 (+3.61), with average classification and retrieval improvements of 2.65 and 3.19 percentage points, respectively. In aggregate, 80% of evaluated model checkpoints see zero-shot gains after RSTeller-based continual pretraining (Ge et al., 2024).

6. Limitations, Impact, and Practitioner Considerations

RSTeller automates annotation after the initial data fetch, eliminating the need for human-authored captions and enabling large-scale VLM pretraining without specialized RS or LLM resources on the researcher's part. The dataset is distributed as WebDataset shards to facilitate distributed and accelerated workflows.

Key limitations include:

• Captions are restricted to a single dominant OSM element per patch, omitting multi-element or holistic scene descriptions.
• Geographic and spectral diversity is limited to 3-band NAIP aerial imagery of the U.S.; absence of multispectral, SAR, LiDAR, or wider global coverage restricts generalization.
• Pretraining protocol is currently limited to CLIP-style contrastive models; no generative or instruction-tuned approaches have yet been explored within RSTeller.

Recommendations for practitioners include integrating multisensor and global sources to mitigate data domain gaps, scaling domain-specific data beyond 1M pairs for stable accuracy, favoring large vision architectures (ViT-L or larger), and leveraging robust initial checkpoints such as those from DataComp for improved continual pretraining (Ge et al., 2024).

RSTeller demonstrates that large, semantically rich RS image–caption datasets can be generated in a fully automated fashion from open geospatial resources and LLMs, enabling reliable zero-shot RS scene understanding improvements and lowering the entry threshold for multimodal RS research. This supports the trajectory toward ever-larger, more diverse RS VLMs and potentially broader downstream impact across geospatial analysis domains (Ge et al., 2024).