Landsat30-AU-VQA: Satellite Visual Q&A Benchmark

• Landsat30-AU-VQA is a large-scale visual question answering dataset featuring 17,725 human-verified samples derived from 30-meter Landsat imagery spanning four missions over Australia from 1988 to 2024.
• The dataset employs a rigorous multi-domain, human-in-the-loop curation pipeline, integrating both fine-tuned models and manual verification to ensure high-quality, long-term remote sensing data.
• Empirical findings reveal that lightweight fine-tuning significantly enhances performance, highlighting distinct model capabilities in tasks like cloud assessment, phenological reasoning, and fine-object detectability.

Searching arXiv for the specified dataset paper and closely related context papers. Tool call: arxiv_search(query="Landsat30-AU-VQA OR Landsat30-AU", max_results=10, sort_by="relevance") Landsat30-AU-VQA is a large-scale, human-verified visual question answering dataset for 30-meter resolution Landsat satellite imagery spanning four missions—Landsat 5, 7, 8, and 9—over Australia from 1988 to 2024. It is the VQA component of the broader Landsat30-AU resource, which was introduced to address a gap in remote sensing vision-language datasets: existing corpora had focused mainly on short-term, high-resolution imagery from a limited number of satellites, whereas Landsat30-AU emphasizes long-term, multi-sensor, low-resolution archives that are central to affordable and bias-robust global monitoring (Ma et al., 5 Aug 2025).

1. Scope and dataset definition

Landsat30-AU-VQA comprises 17,725 VQA samples. Each sample contains a Landsat image tile of size $256 \times 256$ pixels at 30 m GSD, a multiple-choice question, four answer options with one correct answer, and human verification for question and answer quality and correctness (Ma et al., 5 Aug 2025).

The dataset is explicitly oriented toward natural-language interaction with satellite imagery. In that sense, it is not a generic image-question benchmark adapted to remote sensing, but a benchmark constructed around the constraints of Landsat imagery: long temporal coverage, multi-satellite heterogeneity, and the interpretive difficulty of 30-meter spatial resolution. This design choice situates the dataset within a research program concerned with expert workflow acceleration, accessibility for non-specialists, and planet-scale automation, while retaining the specific demands of Earth observation practice.

Landsat30-AU-VQA is paired with a captioning component, Landsat30-AU-Cap, containing 196,262 image-caption pairs. The VQA benchmark therefore sits within a broader dataset architecture in which caption generation and question answering are jointly supported by the same imagery and curation pipeline. This suggests a deliberate attempt to evaluate both free-form descriptive competence and constrained answer selection under the same remote-sensing conditions.

2. Domain coverage and reasoning targets

The dataset covers eight diverse VQA domains relevant to Earth observation. These domains are intended to span low-level perception, spatial and contextual reasoning, counting, object recognition, and robustness to remote-sensing-specific issues (Ma et al., 5 Aug 2025).

Domain	#QAs	Task focus
APR	2,102	Agro-Phenology Reasoning
COA	2,129	Cloud-Occlusion Assessment
DLC	2,479	Dominant Land-Cover Type
FOD	2,000	Fine-Object Detectability
MOP	2,418	Macro-Object Presence
NUM	2,244	Numerosity
SRI	2,419	Spatial-Relation Inference
USR	1,934	Urban-Scale Recognition

The examples given for these domains clarify their intended operational semantics. APR includes questions such as “Wet or dry season?”, COA includes “Scene usable despite cloud?”, DLC asks “Main cover type?”, FOD asks about a “Prominent thin structure?”, MOP asks “Which object is visible?”, NUM includes “Water-body count?”, SRI probes relations such as “River vs urban fabric?”, and USR addresses questions such as “Settlement type?” These tasks are not homogeneous. Some are predominantly perceptual, while others require contextual judgments over scene structure or seasonal cues.

The domain balance is methodologically important. The benchmark is not limited to land-cover classification framed as question answering. Instead, it targets several distinct error modes known to be difficult in remote sensing VLMs: cloud-related usability judgments, thin-object detectability at 30 m GSD, numerosity, and phenological interpretation. A plausible implication is that aggregate benchmark performance can conceal materially different capabilities across subdomains, which is why per-category accuracy is reported.

3. Curation pipeline and quality controls

Landsat30-AU-VQA was curated through a rigorous, iterative and human-in-the-loop curation pipeline. The pipeline begins with imagery sampled across decades, spatially stratified, and filtered for cloud cover using both metadata and Qwen2.5-VL-7B classification. It then introduces fine-tuned modules for domain-specific subtasks, and finally generates VQA items from captions using GPT-4.1, followed by manual review for correctness, ambiguity, and adversarially challenging options (Ma et al., 5 Aug 2025).

Several seed resources are reported for the intermediate stages: 2,722 region labels, 1,005 captions, and 9,440 caption-sentence labels, all human-verified. Fine-tuning was performed on GPT-4o for region tasks and Qwen2.5-VL-7B for caption review. Model selection during these stages used Subset Accuracy, Jaccard, Precision, Recall, F1, nDCG, and LRAP. One reported comparison gives Subset Accuracy for region classification as 0.28 for GPT-4o fine-tuned versus 0.10 for Qwen without fine-tune.

The pipeline also includes several dataset-specific controls. OSM tags were mapped to 25 land-use categories, explicitly chosen to be compatible with 30 m GSD. Caption refinement proceeds through initial generation, missing-object/connection enrichment with Qwen2.5-VL-7B, and pruning by a hallucination filter. Human reviewers are described as fixing factual errors and ambiguous phrasings, replacing weak distractors, adversarially enhancing difficulty, and discarding or modifying items as necessary.

One of the most consequential protocol choices is that no geolocation or temporal metadata is exposed to models during evaluation to avoid answer leakage. That decision constrains the benchmark to image-grounded reasoning rather than metadata retrieval. It also narrows the interpretation of model success: strong performance cannot be attributed to direct access to location or timestamp fields.

4. Evaluation protocol and benchmark design

The benchmark uses a 15% holdout from Landsat30-AU-VQA as the test set. For each test item, a model receives the image, the question text, and the list of four candidate answers. Scoring is by exact match between model output and the correct answer, after case and whitespace trimming (Ma et al., 5 Aug 2025).

The primary metric is VQA Accuracy, defined as the proportion of test examples where the model correctly selects the answer. The benchmark also reports per-category accuracy for diagnostic analysis. Captioning is evaluated separately on a different holdout set using BLEU-4, SPIDEr, BERT-F1, 1-CHAIR-s/i for hallucination, and average caption length.

The models evaluated include both specialized and general-purpose VLMs. The reported set comprises EarthDial, RS-LLaVA, MiMo, GLM-V, Qwen2.5-VL-7B, Llama, Gemma3, and LLaVA, along with fine-tuned variants Qwen-ft and Llama-ft. The fine-tuned versions use LoRA adapters trained on 15% Landsat30-AU data. Additional implementation details include LoRA-based efficient fine-tuning, single epoch, quantized backbone, and rank-64 adapters.

This protocol makes two aspects of the benchmark unusually clear. First, the task is not open-ended generative VQA but multiple-choice selection under strict exact-match evaluation. Second, because metadata leakage is excluded and answers are constrained to four options, performance differences can be interpreted primarily in terms of visual grounding, question understanding, and adaptation to Landsat imagery rather than free-form response style.

5. Empirical findings

The central empirical finding is that off-the-shelf models struggle to understand satellite imagery, including models marketed or developed for remote sensing. The paper reports that the open-source remote-sensing VLM EarthDial achieves 0.07 SPIDEr in captioning and 0.48 VQA accuracy, while the more detailed benchmark table gives 0.4829 overall accuracy for EarthDial on VQA (Ma et al., 5 Aug 2025).

Among non-fine-tuned models, general VLMs outperform specialized ones in aggregate. Reported overall VQA accuracies include Qwen: 0.7428 and Gemma3: 0.7356. EarthDial is reported as catastrophic on some categories, including COA: 0.10 and APR: 0.23. RS-LLaVA is also described as underperforming on core reasoning, with USR and SRI in the 0.10–0.26 range. At the same time, MiMo and GLM-V are described as showing strengths in NUM and MOP, although their captions are reported to be verbose and hallucination-prone.

Lightweight fine-tuning changes the picture substantially. Fine-tuning Qwen2.5-VL-7B on Landsat30-AU improves captioning from 0.11 to 0.31 SPIDEr and increases overall VQA accuracy from 0.74 to 0.87; the more detailed table reports Qwen-ft: 0.8710. Llama-ft also improves but less dramatically, reaching 0.73 overall VQA accuracy. The per-category profile for Qwen-ft is especially notable: it achieves the best per-category accuracy in 6 of 8 domains, reaches 1.00 on FOD, and raises APR to 0.70 from 0.30.

A compact view of representative results is shown below.

Model	Overall VQA accuracy	Selected observations
EarthDial	0.48	COA 0.10, APR 0.23
Qwen	0.74	Strong aggregate baseline
Qwen-ft	0.87	Best in 6 of 8 domains; FOD 1.00

These results support two benchmark-level conclusions. The first is that remote-sensing specialization alone does not guarantee strong performance on long-term, low-resolution Landsat imagery. The second is that fine-tuning on domain-appropriate, human-verified, multi-temporal VQA data is essential for robust performance on Landsat imagery. A common misconception is that a VLM trained for remote sensing should transfer cleanly across satellite regimes; Landsat30-AU-VQA provides counterevidence, especially for phenology, cloud assessment, and urban-scale interpretation.

6. Position within Landsat-based remote sensing research

Landsat30-AU-VQA is defined partly by the properties of the Landsat archive itself: 30-meter resolution, multi-satellite coverage, and a temporal span of more than 36 years over Australia. In comparison to earlier vision-language corpora described in the paper, it emphasizes 4 Landsat satellites, 36 years (1988–2024), and a scale of 17,725 human-verified VQA examples, paired with 196,262 captions (Ma et al., 5 Aug 2025).

This positioning matters because 30-meter Landsat imagery occupies a distinct niche in Earth observation. It is lower resolution than many contemporary VLM datasets, but it is also historically extensive and comparatively affordable for long-term monitoring. Related Landsat-based work outside the vision-language domain illustrates why this matters. For example, a 30 m resolution global annual burned area mapping study based on Landsat images and Google Earth Engine emphasized Landsat’s role in scalable annual monitoring and noted regional relevance for Australia, where such products can complement and extend regional fire products such as the Australian Fire Scars (Long et al., 2018). This suggests that a VQA benchmark grounded in Landsat is aligned with practical remote-sensing workflows that depend on long-baseline archives rather than only high-resolution contemporary imagery.

The benchmark therefore occupies a specific methodological role. It is not merely a dataset for measuring generic multimodal competence; it is a testbed for whether VLMs can operate under the information bottlenecks, ambiguities, and temporal breadth that characterize operational Landsat analysis. Its strongest contribution is likely the combination of human verification, multi-domain task design, strict evaluation, and explicit anti-leakage controls. A plausible implication is that future remote-sensing VLM research will need to distinguish more sharply between performance on high-resolution short-term imagery and performance on long-term, low-resolution, multi-sensor archives, because Landsat30-AU-VQA shows that these regimes are not interchangeable.