---
title: 'Landsat30-AU-VQA: Satellite Visual Q&A Benchmark'
url: https://www.emergentmind.com/topics/landsat30-au-vqa
type: topic
---

# Landsat30-AU-VQA: Satellite Visual Q&A Benchmark

Searching arXiv for the specified dataset paper and closely related context papers.
Tool call: arxiv_search(query="Landsat30-AU-VQA OR Landsat30-AU", max_results=10, sort_by="relevance")
Landsat30-AU-VQA is a large-scale, human-verified visual question answering dataset for 30-meter resolution Landsat satellite imagery spanning four missions—Landsat 5, 7, 8, and 9—over Australia from 1988 to 2024. It is the VQA component of the broader Landsat30-AU resource, which was introduced to address a gap in remote sensing vision-language datasets: existing corpora had focused mainly on short-term, high-resolution imagery from a limited number of satellites, whereas Landsat30-AU emphasizes long-term, multi-sensor, low-resolution archives that are central to affordable and bias-robust global monitoring [2508.03127].

## 1. Scope and dataset definition

Landsat30-AU-VQA comprises **17,725** VQA samples. Each sample contains a **Landsat image tile** of size \(256 \times 256\) pixels at **30 m GSD**, a **multiple-choice question**, **four answer options** with **one correct answer**, and **human verification** for question and answer quality and correctness [2508.03127].

The dataset is explicitly oriented toward natural-language interaction with satellite imagery. In that sense, it is not a generic image-question benchmark adapted to remote sensing, but a benchmark constructed around the constraints of Landsat imagery: long temporal coverage, multi-satellite heterogeneity, and the interpretive difficulty of 30-meter spatial resolution. This design choice situates the dataset within a research program concerned with expert workflow acceleration, accessibility for non-specialists, and planet-scale automation, while retaining the specific demands of Earth observation practice.

Landsat30-AU-VQA is paired with a captioning component, **Landsat30-AU-Cap**, containing **196,262 image-caption pairs**. The VQA benchmark therefore sits within a broader dataset architecture in which caption generation and question answering are jointly supported by the same imagery and curation pipeline. This suggests a deliberate attempt to evaluate both free-form descriptive competence and constrained answer selection under the same remote-sensing conditions.

## 2. Domain coverage and reasoning targets

The dataset covers **eight diverse VQA domains** relevant to Earth observation. These domains are intended to span low-level perception, spatial and contextual reasoning, counting, object recognition, and robustness to remote-sensing-specific issues [2508.03127].

| Domain | #QAs | Task focus |
|---|---:|---|
| APR | 2,102 | Agro-Phenology Reasoning |
| COA | 2,129 | Cloud-Occlusion Assessment |
| DLC | 2,479 | Dominant Land-Cover Type |
| FOD | 2,000 | Fine-Object Detectability |
| MOP | 2,418 | Macro-Object Presence |
| NUM | 2,244 | Numerosity |
| SRI | 2,419 | Spatial-Relation Inference |
| USR | 1,934 | Urban-Scale Recognition |

The examples given for these domains clarify their intended operational semantics. **APR** includes questions such as “Wet or dry season?”, **COA** includes “Scene usable despite cloud?”, **DLC** asks “Main cover type?”, **FOD** asks about a “Prominent thin structure?”, **MOP** asks “Which object is visible?”, **NUM** includes “Water-body count?”, **SRI** probes relations such as “River vs urban fabric?”, and **USR** addresses questions such as “Settlement type?” These tasks are not homogeneous. Some are predominantly perceptual, while others require contextual judgments over scene structure or seasonal cues.

The domain balance is methodologically important. The benchmark is not limited to land-cover classification framed as question answering. Instead, it targets several distinct error modes known to be difficult in remote sensing VLMs: cloud-related usability judgments, thin-object detectability at 30 m GSD, numerosity, and phenological interpretation. A plausible implication is that aggregate benchmark performance can conceal materially different capabilities across subdomains, which is why per-category accuracy is reported.

## 3. Curation pipeline and quality controls

Landsat30-AU-VQA was curated through a **rigorous, iterative and human-in-the-loop curation pipeline**. The pipeline begins with imagery sampled across decades, spatially stratified, and filtered for cloud cover using both metadata and **Qwen2.5-VL-7B classification**. It then introduces fine-tuned modules for domain-specific subtasks, and finally generates VQA items from captions using **GPT-4.1**, followed by manual review for correctness, ambiguity, and adversarially challenging options [2508.03127].

Several seed resources are reported for the intermediate stages: **2,722 region labels**, **1,005 captions**, and **9,440 caption-sentence labels**, all human-verified. Fine-tuning was performed on **GPT-4o** for region tasks and **Qwen2.5-VL-7B** for caption review. Model selection during these stages used **Subset Accuracy**, **Jaccard**, **Precision**, **Recall**, **F1**, **nDCG**, and **LRAP**. One reported comparison gives **Subset Accuracy for region classification** as **0.28** for **GPT-4o fine-tuned** versus **0.10** for **Qwen without fine-tune**.

The pipeline also includes several dataset-specific controls. **OSM tags** were mapped to **25 land-use categories**, explicitly chosen to be compatible with **30 m GSD**. Caption refinement proceeds through **initial generation**, **missing-object/connection enrichment** with **Qwen2.5-VL-7B**, and **pruning by a hallucination filter**. Human reviewers are described as fixing factual errors and ambiguous phrasings, replacing weak distractors, adversarially enhancing difficulty, and discarding or modifying items as necessary.

One of the most consequential protocol choices is that **no geolocation or temporal metadata is exposed to models during evaluation to avoid answer leakage**. That decision constrains the benchmark to image-grounded reasoning rather than metadata retrieval. It also narrows the interpretation of model success: strong performance cannot be attributed to direct access to location or timestamp fields.

## 4. Evaluation protocol and benchmark design

The benchmark uses a **15% holdout** from Landsat30-AU-VQA as the test set. For each test item, a model receives the **image**, the **question text**, and the **list of four candidate answers**. Scoring is by **exact match** between model output and the correct answer, after **case and whitespace trimming** [2508.03127].

The primary metric is **VQA Accuracy**, defined as the proportion of test examples where the model correctly selects the answer. The benchmark also reports **per-category accuracy** for diagnostic analysis. Captioning is evaluated separately on a different holdout set using **BLEU-4**, **SPIDEr**, **BERT-F1**, **1-CHAIR-s/i** for hallucination, and **average caption length**.

The models evaluated include both specialized and general-purpose VLMs. The reported set comprises **EarthDial, RS-LLaVA, MiMo, GLM-V, Qwen2.5-VL-7B, Llama, Gemma3, and LLaVA**, along with fine-tuned variants **Qwen-ft** and **Llama-ft**. The fine-tuned versions use **LoRA adapters trained on 15% Landsat30-AU data**. Additional implementation details include **LoRA-based efficient fine-tuning**, **single epoch**, **quantized backbone**, and **rank-64 adapters**.

This protocol makes two aspects of the benchmark unusually clear. First, the task is not open-ended generative VQA but multiple-choice selection under strict exact-match evaluation. Second, because metadata leakage is excluded and answers are constrained to four options, performance differences can be interpreted primarily in terms of visual grounding, question understanding, and adaptation to Landsat imagery rather than free-form response style.

## 5. Empirical findings

The central empirical finding is that **off-the-shelf models struggle to understand satellite imagery**, including models marketed or developed for remote sensing. The paper reports that the open-source remote-sensing VLM **EarthDial** achieves **0.07 SPIDEr** in captioning and **0.48** VQA accuracy, while the more detailed benchmark table gives **0.4829 overall accuracy** for EarthDial on VQA [2508.03127].

Among non-fine-tuned models, **general VLMs outperform specialized ones in aggregate**. Reported overall VQA accuracies include **Qwen: 0.7428** and **Gemma3: 0.7356**. EarthDial is reported as **catastrophic** on some categories, including **COA: 0.10** and **APR: 0.23**. **RS-LLaVA** is also described as underperforming on core reasoning, with **USR** and **SRI** in the **0.10–0.26** range. At the same time, **MiMo** and **GLM-V** are described as showing strengths in **NUM** and **MOP**, although their captions are reported to be **verbose and hallucination-prone**.

Lightweight fine-tuning changes the picture substantially. Fine-tuning **Qwen2.5-VL-7B** on Landsat30-AU improves captioning **from 0.11 to 0.31 SPIDEr** and increases overall VQA accuracy **from 0.74 to 0.87**; the more detailed table reports **Qwen-ft: 0.8710**. **Llama-ft** also improves but less dramatically, reaching **0.73** overall VQA accuracy. The per-category profile for Qwen-ft is especially notable: it achieves the **best per-category accuracy in 6 of 8 domains**, reaches **1.00** on **FOD**, and raises **APR** to **0.70** from **0.30**.

A compact view of representative results is shown below.

| Model | Overall VQA accuracy | Selected observations |
|---|---:|---|
| EarthDial | 0.48 | COA 0.10, APR 0.23 |
| Qwen | 0.74 | Strong aggregate baseline |
| Qwen-ft | 0.87 | Best in 6 of 8 domains; FOD 1.00 |

These results support two benchmark-level conclusions. The first is that remote-sensing specialization alone does not guarantee strong performance on long-term, low-resolution Landsat imagery. The second is that **fine-tuning on domain-appropriate, human-verified, multi-temporal VQA data is essential for robust performance on Landsat imagery**. A common misconception is that a VLM trained for remote sensing should transfer cleanly across satellite regimes; Landsat30-AU-VQA provides counterevidence, especially for phenology, cloud assessment, and urban-scale interpretation.

## 6. Position within Landsat-based remote sensing research

Landsat30-AU-VQA is defined partly by the properties of the Landsat archive itself: **30-meter resolution**, **multi-satellite coverage**, and a temporal span of **more than 36 years** over Australia. In comparison to earlier vision-language corpora described in the paper, it emphasizes **4 Landsat satellites**, **36 years (1988–2024)**, and a scale of **17,725 human-verified VQA** examples, paired with **196,262 captions** [2508.03127].

This positioning matters because 30-meter Landsat imagery occupies a distinct niche in Earth observation. It is lower resolution than many contemporary VLM datasets, but it is also historically extensive and comparatively affordable for long-term monitoring. Related Landsat-based work outside the vision-language domain illustrates why this matters. For example, a 30 m resolution global annual burned area mapping study based on Landsat images and Google Earth Engine emphasized Landsat’s role in scalable annual monitoring and noted regional relevance for Australia, where such products can complement and extend regional fire products such as the Australian Fire Scars [1805.02579]. This suggests that a VQA benchmark grounded in Landsat is aligned with practical remote-sensing workflows that depend on long-baseline archives rather than only high-resolution contemporary imagery.

The benchmark therefore occupies a specific methodological role. It is not merely a dataset for measuring generic multimodal competence; it is a testbed for whether VLMs can operate under the information bottlenecks, ambiguities, and temporal breadth that characterize operational Landsat analysis. Its strongest contribution is likely the combination of **human verification**, **multi-domain task design**, **strict evaluation**, and **explicit anti-leakage controls**. A plausible implication is that future remote-sensing VLM research will need to distinguish more sharply between performance on high-resolution short-term imagery and performance on long-term, low-resolution, multi-sensor archives, because Landsat30-AU-VQA shows that these regimes are not interchangeable.

Source: https://www.emergentmind.com/topics/landsat30-au-vqa