Landsat30-AU-Cap: 30-m Landsat Captioning

• Landsat30-AU-Cap is a vision-language component pairing 30-m resolution Landsat imagery with detailed captions to support remote sensing model training.
• It is constructed through a multi-stage human-in-the-loop pipeline that refines imagery, OSM tags, and DEA land cover data for enhanced spatial and temporal accuracy.
• The dataset serves as a benchmark for captioning tasks on multi-sensor, multi-decadal imagery, illustrating trade-offs between semantic quality and hallucination resistance.

Landsat30-AU-Cap is the captioning component of Landsat30-AU, a vision-language dataset built from 30 m resolution Landsat imagery over Australia. It contains 196,262 image-caption pairs derived from imagery collected by Landsat 5, 7, 8, and 9 across 1988 to 2024, and it was introduced to support training and evaluation of models that can describe, understand, and interact in natural language with Landsat imagery. The dataset is framed as a response to a specific gap in remote-sensing vision-language resources: most existing datasets emphasize higher-resolution imagery, shorter temporal windows, fewer Landsat sensors, or text supervision that is poorly aligned with what is actually visible at Landsat scale (Ma et al., 5 Aug 2025).

1. Definition and role within Landsat30-AU

Landsat30-AU has two components: Landsat30-AU-Cap, with 196,262 image-caption pairs, and Landsat30-AU-VQA, with 17,725 human-verified visual question answering samples. The two parts are complementary. Landsat30-AU-Cap provides descriptive natural-language supervision for free-form scene description, whereas Landsat30-AU-VQA probes structured question answering and reasoning across eight remote sensing domains. The paper explicitly states that Landsat30-AU-Cap “supports training and evaluation of captioning models on real-world, multi-sensor, multi-temporal satellite imagery,” while Landsat30-AU-VQA is intended to “capture common reasoning challenges in low-resolution imagery” (Ma et al., 5 Aug 2025).

The dataset’s rationale is tightly coupled to the semantics of 30 m Landsat imagery. The paper argues that object-centric captioning conventions from higher-resolution remote-sensing datasets do not transfer cleanly to Landsat, because many commonly named objects in those resources—such as cars, small rooftops, and lane markings—are not visible at this spatial scale. Landsat30-AU-Cap was therefore designed around Landsat-native resolution awareness, multiple Landsat satellites, long temporal coverage, and text supervision appropriate for image-language alignment. A common misconception is that remote-sensing captioning is largely resolution-agnostic; the dataset is explicitly constructed to reject that assumption.

2. Data composition, sampling, and geographic-temporal scope

The imagery is sourced from the Digital Earth Australia (DEA) Analysis Ready Data (ARD) archive. Samples are represented as $256 \times 256$ RGB tiles at 30-meter ground sample distance, produced in the main text from Bands 4/3/2 (Red/Green/Blue) and, in the appendix, from DEA ARD nbart_red, nbart_green, nbart_blue bands as true-color RGB patches. At 30 m per pixel, each tile corresponds to a ground footprint of roughly $7{,}680 \times 7{,}680\ \mathrm{m}^2$. Coverage is restricted to Australia, with the paper noting stronger spatial density in eastern and southwestern Australia, consistent with the country’s main populated and agriculturally significant regions.

Aspect	Value	Notes
Caption corpus size	196,262 image-caption pairs	Final released Landsat30-AU-Cap
Sensor coverage	Landsat 5, 7, 8, 9	Multi-sensor Landsat archive
Spatial representation	$256 \times 256$ RGB tiles at 30 m GSD	Roughly $7{,}680 \times 7{,}680\ \mathrm{m}^2$ footprint
Geographic scope	Australia only	Density concentrated in eastern and southwestern regions
Temporal span	1988–2024	Described as 36 years and “more than 36 years”
Seed verified caption set	1,005 pairs	Split into 70% train / 15% validation / 15% test
Corpus statistics	Vocabulary 4,405; average caption length 165.4 words; MSTTR 0.82	Reported for Landsat30-AU caption corpus

The sampling strategy is deliberate rather than opportunistic. Australia was partitioned into a static grid of approximately 2,500 AOIs. For each AOI, the authors aimed to select one image per year, and to encourage seasonal diversity the search began in a different quarter each year according to year modulo 4; the appendix gives 2001, beginning in Q2 (April 1st), as an example. The first image passing the initial quality filter was selected. Before final filtering, Stage 1 produced over 400,000 candidate RGB tiles; after additional filtering and selection, the dataset was reduced to 196,262 high-quality Landsat images, which are the same images used for caption generation (Ma et al., 5 Aug 2025).

Initial filtering retained only patches marked as at least $99.5\%$ cloud-free by DEA ARD metadata. Manual inspection showed that this criterion alone was insufficient, because many cloudy scenes still passed and many retained scenes were semantically uninformative. A second filtering stage used Qwen2.5-VL-7B to classify each patch as “cloudy” or “clear,” with “cloudy” defined broadly enough to include severe sensor artifacts, striping, missing data, over-exposure, and other conditions that obstruct useful interpretation.

From the construction process, each sample is associated with at least the Landsat image tile, geographic location or geohash, capture time or acquisition date, satellite identifier, region-level land-cover labels, mapped OSM land-use categories, and a generated or reviewed caption. The paper explicitly notes that the draft imagery metadata included geohash, capture time, and satellite metadata, and that Landsat30-AU retains precise geographic coordinates and acquisition dates. It does not provide a formal JSON schema or file-format specification, so anything beyond this field set would be speculative.

3. Human-in-the-loop construction pipeline

Landsat30-AU-Cap was built with a three-stage human-in-the-loop bootstrapped pipeline using generic VLMs, iterative refinement, and human verification. The first stage prepares imagery and auxiliary metadata. After RGB tile extraction and quality filtering, the pipeline extracts OpenStreetMap tags within each tile footprint. Because raw OSM labels are noisy, inconsistent, and often too fine-grained for 30 m imagery, they are normalized into a controlled vocabulary of 25 distinct land-use categories appropriate for Landsat-scale interpretation. The paper provides explicit mappings such as clinic $\rightarrow$ urban_fabric, river/stream/creek/canal $\rightarrow$ river_stream, farmland/orchard/vineyard $\rightarrow$ cropland, forest/woodland/grassland/shrub $\rightarrow$ natural_vegetation, residential/commercial/industrial/hospital/school $\rightarrow$ urban_fabric, and road/street/highway/railway $7{,}680 \times 7{,}680\ \mathrm{m}^2$0 road_corridor.

The same stage also extracts structured land-cover context from the DEA Land Cover product, an annual pixel-level classification over Australia. For each image tile, the dominant land-cover class is computed for top-left, top-right, bottom-left, bottom-right, center, and the whole tile. The appendix lists seven broad DEA Land Cover categories: Cultivated Terrestrial Vegetation, Natural Terrestrial Vegetation, Natural Aquatic Vegetation, Artificial Surfaces, Natural Bare Surfaces, Water, and No Data. These labels are used as conditioning signals rather than as caption targets.

The second stage fine-tunes three lightweight Landsat-specific modules: region classification, caption generation, and caption review. For region classification, 2,722 tile-region label sets were manually validated, and GPT-4o was fine-tuned with an 80% train / 20% test split using 3 epochs, batch size 4, learning rate multiplier 2, and seed 42. On the fine-tuning set, the resulting model achieved Subset Accuracy 0.278, Jaccard 0.630, Precision 0.768, Recall 0.722, F1 0.727, LRAP 0.826, nDCG 0.917, 1-Hamming-loss 0.783, and 1-ranking-loss 0.705.

For caption generation, the authors manually curated and verified 1,005 image-caption pairs whose text had to reference objects visible at 30 m and align with the acquisition date. GPT-4.1 was fine-tuned on this seed set using a 70% train / 15% validation / 15% test split, 3 epochs, batch size 1, learning rate multiplier 2, and seed 42, with fine-tuning conducted in OpenAI Playground. The captioning prompt conditions on the image, region-level land-cover information, and OSM-derived land-use information, and it explicitly instructs the model to cross-validate sources to avoid hallucinations. If, for example, land cover indicates no water, the model is instructed to ignore water-related OSM tags. This prompt asks for visible water bodies and their relative sizes, dominant land cover by area, artificial areas such as “small town” or “city” if visible, road corridors and directions if visible, spatial references such as top, bottom, left, right, and center, and a summary of the balance between bare and vegetated surfaces. Fine-tuning improved caption quality from BLEU-4 0.152 / SPIDEr 0.438 / BERT-F1 0.901 / 1-CHAIR-s 0.522 / 1-CHAIR-i 0.864 / average length 140 to BLEU-4 0.184 / SPIDEr 0.517 / BERT-F1 0.903 / 1-CHAIR-s 0.473 / 1-CHAIR-i 0.853 / average length 161.

For caption review, the human verification process produced 9,440 image-sentence-decision triplets, each labeled keep or delete. Qwen2.5-VL-7B was then fine-tuned as a sentence-level reviewer using a 70/15/15 split, 3 epochs, batch size 24, Adam, cosine schedule, learning rate $7{,}680 \times 7{,}680\ \mathrm{m}^2$1, warm-up over the first 6% of steps, weight decay $7{,}680 \times 7{,}680\ \mathrm{m}^2$2, Adam $7{,}680 \times 7{,}680\ \mathrm{m}^2$3, bfloat16 precision, and 8 $7{,}680 \times 7{,}680\ \mathrm{m}^2$4 NVIDIA L4-24G GPUs. The model inspects an image and a single caption sentence, returning only “keep” or “delete.”

The third stage applies these modules at scale. A fine-tuned GPT-4.1 captioner first produces an Initial caption from the image, region labels, and OSM tags. Qwen2.5-VL-7B then generates an Extra version by adding either one clearly visible missing object or one missing spatial or contextual relationship; the prompts explicitly forbid mention of absent features. The fine-tuned reviewer removes hallucinated or temporally inconsistent sentences to produce the Reviewed caption. On a held-out reference set, the authors compare Initial, Extra, and Reviewed captions using BLEU-4, SPIDEr, BERTScore-F1, and CHAIR-s/i reported as 1-CHAIR-s and 1-CHAIR-i, concluding that the Reviewed captions provide the best overall balance, specifically SPIDEr 0.517 and 1-CHAIR-i 0.853, and are therefore used throughout the released dataset (Ma et al., 5 Aug 2025).

4. Caption content, style, and semantic coverage

The captions are characterized in the paper as detailed, visually grounded, resolution-aware, and semantically rich. Their intended content includes land cover, land use, water bodies, vegetation versus bare surface balance, artificial surfaces and settlements, roads and corridors, macro-objects and landscape patterns, and spatial relationships. The emphasis is on semantics that are supportable at Landsat scale rather than on fine-grained object inventories that would be more appropriate for sub-meter or few-meter imagery.

The corpus is notably long-form for remote-sensing captioning. Reported linguistic statistics for Landsat30-AU show a vocabulary size of 4,405, average caption length of 165.4 words, and MSTTR of 0.82. During refinement experiments, Reviewed captions averaged around 161 words. The prompts encourage objective, analytical descriptions, including orientation terms such as top-left, center, and bottom-right, discussion of dominance and extent, and mention of both dominant and minor visual elements. The result is natural language that is semi-standardized by prompt instructions but not reduced to rigid templates (Ma et al., 5 Aug 2025).

The paper repeatedly states that captions should mention only what is visible or otherwise supportable at 30 m. This is a core design principle. Visually ungrounded text and temporally mismatched content are removed during review, which directly addresses a recurrent problem in remote-sensing datasets assembled from OSM or other metadata sources that may mention objects too small to resolve or features no longer present at the image date. The captions are also temporally grounded: reviewers checked whether content aligned with the corresponding acquisition date, and the review stage was explicitly used to remove temporally inconsistent text.

Although the paper does not define a formal caption-domain taxonomy for Landsat30-AU-Cap analogous to the eight VQA categories, its prompts, mappings, and examples indicate broad coverage of agriculture or cropland, natural vegetation, water bodies, urban fabric or settlements, road corridors, natural bare surfaces or arid landscapes, wetlands, rivers and streams, and some industrial or extractive features. Human verification examples include the sentences “A golf course appears in the image. Decision: Keep.”, “A small dam in the lower right. Decision: Keep.”, and “Grid of white oil or gas well pads. Decision: Keep.” The word cloud reported for Landsat30-AU-Cap is dominated by vegetation, bare surface, and water, which is consistent with the dataset’s landscape-scale emphasis.

The paper does not report any caption-domain balancing procedure. A plausible implication is that topic frequency follows the AOI sampling design and the underlying Australian imagery distribution rather than an explicit semantic equalization scheme.

5. Benchmark protocol, model performance, and fine-tuning behavior

Landsat30-AU-Cap is also a benchmark for image captioning on Landsat imagery. In evaluation, models are given only the raw image and not the auxiliary land-cover or land-use metadata used during dataset construction. The benchmark prompt states that each pixel is 30 meters and asks the model to describe land covers, features, surface types, and spatial relationships in detailed, domain-specific language, including both dominant and minor elements and explicit orientation terms, while basing descriptions only on observable features. This makes the benchmark a test of learned Landsat-specific visual understanding rather than metadata exploitation (Ma et al., 5 Aug 2025).

The captioning benchmark uses BLEU-4, SPIDEr, BERTScore-F1, 1-CHAIR-s, 1-CHAIR-i, and Average Caption Length. The paper describes BLEU-4 as a 4-gram overlap measure, SPIDEr as a composite of SPICE and CIDEr intended to capture semantic alignment and informative phrase overlap, BERTScore-F1 as contextual-embedding-based semantic similarity, and CHAIR-s / CHAIR-i as sentence- and instance-level hallucination rates, reported in complemented form so that higher 1-CHAIR is better.

Eight base VLMs are evaluated: specialized models EarthDial (4B), RS-LLaVA (7B), MiMo (7B), and GLM-V (9B); general models Qwen (7B), LLaVA (8B), Llama (11B), and Gemma 3 (12B); and fine-tuned variants Qwen-ft (7B) and Llama-ft (11B). The reported captioning results are exact. EarthDial achieves BLEU-4 0.0210, SPIDEr 0.0726, BERTScore-F1 0.8379, 1-CHAIR-s 0.5920, 1-CHAIR-i 0.8197, and average caption length 140. RS-LLaVA reaches SPIDEr 0.2095, the highest among the specialized off-the-shelf models. Among general off-the-shelf models, Llama performs best on SPIDEr at 0.1695. These results motivate the paper’s conclusion that off-the-shelf VLMs struggle to understand Landsat imagery.

The strongest gains come from lightweight fine-tuning. The benchmark fine-tunes Qwen2.5-VL-7B and the Llama-3.2 vision-LLM using LoRA in the main text and QLoRA in the appendix, with 15% of the respective training data for each task. The appendix specifies a 4-bit NormalFloat quantized backbone, rank-64 LoRA adapters, $7{,}680 \times 7{,}680\ \mathrm{m}^2$5, dropout 0.05, insertion into all attention projections (q, k, v, o), the gated MLP stack (gate, up, down), and the cross-modal vision projector, trained for 1 epoch with AdamW, learning rate $7{,}680 \times 7{,}680\ \mathrm{m}^2$6, cosine annealing, warm-up fraction 6%, weight decay $7{,}680 \times 7{,}680\ \mathrm{m}^2$7, and 8 $7{,}680 \times 7{,}680\ \mathrm{m}^2$8 NVIDIA L4-24G GPUs.

For Qwen, fine-tuning improves SPIDEr from 0.1114 to 0.3054, BLEU-4 from 0.0350 to 0.1395, BERTScore-F1 from 0.8693 to 0.8935, 1-CHAIR-i from 0.7959 to 0.8549, and average caption length from 124 to 157. For Llama, fine-tuning improves SPIDEr from 0.1695 to 0.2767 and BERTScore-F1 from 0.8800 to 0.8914, but 1-CHAIR-s declines from 0.5483 to 0.5224 and 1-CHAIR-i from 0.8296 to 0.8016, which the paper interprets as a trade-off between semantic quality and hallucination resistance. The abstract rounds the Qwen result as an improvement from 0.11 to 0.31 SPIDEr. A careful reading suggests that the manually verified 1,005-pair seed set underlies the captioning benchmark, with evaluation conducted on its held-out portion.

6. Limitations, caveats, and practical use

Several caveats are explicit or strongly implied. The most direct is geographic restriction: Landsat30-AU-Cap covers Australia only. Ecological, land-use, and climatic patterns are therefore region-specific, and the spatial distribution is denser in eastern and southwestern Australia, which likely biases the corpus toward populated and agriculturally significant environments. A second caveat is residual quality risk from cloud contamination and sensor artifacts. The need for a second Qwen-based filtering pass after the $7{,}680 \times 7{,}680\ \mathrm{m}^2$9 ARD cloud-free threshold indicates that metadata filtering alone was insufficient.

A third caveat concerns metadata noise and temporal mismatch. The pipeline uses OSM tags and DEA Land Cover as supporting context, but OSM may be spatially or temporally mismatched, may refer to objects smaller than Landsat can resolve, and DEA Land Cover is an annual summary product rather than a per-image annotation. The construction procedure attempts to mitigate these problems by cross-validating modalities and deleting unsupported sentences, but it does not eliminate them. A fourth caveat is dependence on generic VLM bootstrapping: the caption corpus is not manually authored from scratch, and the final language may therefore retain some stylistic or semantic biases of GPT-4.1 and Qwen2.5-VL-7B.

Human verification is a major part of the quality-control story, but the paper does not report the number of annotators, their expertise levels, inter-annotator agreement, Cohen’s kappa, or a formal adjudication process. Reviewers judged captions sentence by sentence with binary keep/delete labels, using the criterion of visual verifiability and, when necessary, consulting pansharpened 15 m imagery and third-party services such as Google Earth or Google Maps. This supports the dataset’s claim to visual grounding, but not via formal reliability statistics. The benchmark’s image-only setup is also a double-edged design choice: it yields a fairer test of Landsat understanding, but it may understate performance achievable in practical systems that are allowed to use metadata.

The dataset and code are reported as available at https://github.com/papersubmit1/landsat30-au. The paper cites upstream licensing for DEA Land Cover as Creative Commons Attribution 4.0 Licence and for OpenStreetMap as ODbL, but it does not specify the license of the final Landsat30-AU repository itself. That omission matters for downstream reuse. Overall, the dataset’s main significance lies in pairing multi-sensor, multi-decadal, 30 m Landsat imagery with long-form, reviewed, resolution-aware captions, while its main caveat is that the supervision remains bootstrapped, region-specific, and only partially characterized from an annotation-governance perspective (Ma et al., 5 Aug 2025).