GeoMMBench: Multimodal Geoscience QA Benchmark

• GeoMMBench is a comprehensive multimodal QA benchmark designed for expert geoscience tasks, integrating diverse sensor modalities and disciplinary insights.
• It features image-dependent, expert-written multiple-choice questions spanning remote sensing, photogrammetry, GIS, and GNSS to rigorously assess visual grounding and reasoning.
• Accompanied by the GeoMMAgent framework, it highlights model deficiencies in domain knowledge and demonstrates improved performance through tool-augmented strategies.

GeoMMBench is a comprehensive multimodal question-answering benchmark for geoscience and remote sensing, introduced together with GeoMMAgent to evaluate expert-level multimodal intelligence under the joint pressures of wide-ranging disciplinary knowledge, heterogeneous sensor modalities, and a fragmented spectrum of tasks. It is organized as an image-based, expert-written multiple-choice benchmark spanning remote sensing, photogrammetry, GIS, and GNSS, and it is designed to reveal systematic deficiencies in domain knowledge, perceptual grounding, and reasoning among open-source and proprietary models (Xiao et al., 10 Apr 2026).

1. Scope, definition, and organizing axes

GeoMMBench is a multimodal multiple-choice QA benchmark whose instances consist of an image, a natural-language question, multiple-choice options, and one correct answer. The benchmark is explicitly image-dependent: during curation, any item answerable from text alone is fixed or discarded. Its target is not generic visual question answering, but expert-level geospatial interpretation across four disciplines, six sensor modalities, and six task types (Xiao et al., 10 Apr 2026).

The benchmark is structured along three orthogonal axes.

Axis	Categories	Role
Disciplines	RS, Ph., GIS, GNSS	Disciplinary breadth
Sensor modalities	Optical, HSI, SAR, LiDAR, DEM, Thermal	Heterogeneous sensing
Task spectrum	Pri., Per., Spa., Qua., Tim., App.	From theory to applications

The disciplinary axis covers remote sensing, photogrammetry, geographic information systems, and global navigation satellite systems. The modality axis includes optical RGB imagery, multispectral or hyperspectral imagery treated jointly as HSI, SAR, LiDAR, DEM, and thermal imagery. The task axis spans theoretical principles, perception, spatial reasoning, quality-related tasks such as correction and inspection, time-series analysis, and application-oriented interpretation (Xiao et al., 10 Apr 2026).

The benchmark is also described through a four-level task hierarchy. The first level concerns theoretical knowledge across disciplines. The second level targets low-level perception and quality. The third level targets mid-level geospatial recognition, including scene and object interpretation, relations, and temporal changes. The fourth level concerns high-level applications such as environmental monitoring, agriculture, economic analysis, and mapping products. This hierarchy makes GeoMMBench a benchmark of multimodal geospatial cognition rather than a narrow remote-sensing image classification suite (Xiao et al., 10 Apr 2026).

2. Dataset composition and curation

GeoMMBench contains 1,053 questions, split into a validation set of 37 questions and a test set of 1,016 questions. All questions are image-based, expert-written, multiple-choice items with exactly one correct answer. The source materials come from textbooks, online resources, and academic literature, and the curation process is carried out by PhD-level researchers and doctoral students in geoscience and remote sensing (Xiao et al., 10 Apr 2026).

Its visual coverage extends beyond conventional satellite imagery. The benchmark includes optical satellite and aerial imagery, MSI or HSI imagery, SAR, LiDAR, DEM, and thermal data. It also includes non-sensor visual forms such as maps, remote-sensing mapping products, charts, plots, diagrams, tables, and mathematical notation. That heterogeneity is deliberate: the benchmark is meant to test not only scene perception but also the ability to ground specialized concepts in technical visual representations such as spectral plots, electromagnetic-spectrum diagrams, thematic maps, and index products (Xiao et al., 10 Apr 2026).

Question construction follows a double expert review pipeline. Items that permit text-only inference are removed or revised, and ambiguous wording, misleading distractors, or incorrect answers are corrected. The distractors are crafted to remain geoscientifically plausible, so the task is not reducible to eliminating obviously irrelevant options. The result is a benchmark that stresses visually grounded disciplinary knowledge rather than generic commonsense or language priors (Xiao et al., 10 Apr 2026).

3. Evaluation protocol and baseline performance

GeoMMBench is evaluated in a zero-shot setting. Models use their default multiple-choice prompting, answer extraction is rule-based via regular expressions, and invalid or ambiguous outputs are counted as incorrect. The primary metric is micro-averaged accuracy over multiple-choice questions:

$$\text{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i).$$

The evaluation is implemented with the LMMs-Eval toolkit (Xiao et al., 10 Apr 2026).

The paper evaluates 36 open-source and proprietary LLMs. The random-guessing baseline is 25.2%, and the human expert score on the validation set is 86.5%. Among standalone closed-source MLLMs, Gemini-1.5-Pro achieves 70.7% test accuracy, Gemini-2.0-Flash 70.1%, GPT-4o 68.5%, GPT-o1 65.8%, and GPT-4V 61.0%. Among open-source MLLMs, Qwen-3-VL-30B reaches 66.7%, Qwen-3-VL-8B 65.8%, and Qwen-2.5-VL-7B 62.5%. A text-only GPT-4 baseline obtains 31.2%, indicating that domain knowledge without visual grounding is insufficient for the benchmark (Xiao et al., 10 Apr 2026).

A concise summary of the headline numbers is as follows.

System	Setting	Accuracy
Human expert	Validation	86.5%
Gemini-1.5-Pro	Test	70.7%
Qwen-3-VL-30B	Test	66.7%
GPT-4 Text	Test	31.2%
GeoMMAgent	Test	88.4%

Remote-sensing-specialized MLLMs perform substantially below the strongest general-purpose models: VHM reaches 39.1%, GeoChat 35.4%, TeoChat 31.1%, LHRS-Bot 31.1%, and SkySenseGPT 36.5%. This pattern is used in the paper to argue that current RS instruction tuning pipelines remain narrow and do not transfer to the broader knowledge-and-reasoning demands of GeoMMBench (Xiao et al., 10 Apr 2026).

4. Error patterns and what GeoMMBench measures

The benchmark’s central empirical finding is that current multimodal models exhibit systematic deficiencies in domain knowledge, perceptual grounding, and reasoning. The paper emphasizes visual-language misalignment as a recurring failure mode: models may correctly describe microwave properties in a purely textual setting, yet fail to identify a marked microwave band in an electromagnetic-spectrum diagram when the same knowledge must be visually grounded (Xiao et al., 10 Apr 2026).

Spatial reasoning is another major weakness. GeoMMBench includes tasks that require interpreting compass directions, scale bars, coordinates, relative positions, and approximate distances. Even strong models degrade on these tasks, especially when reasoning goes beyond trivial left-right relations. The same section of the paper reports confusion across sensor modalities: models misclassify HSI, thermal, SAR, optical, and DEM inputs, and they frequently misinterpret SAR-specific phenomena such as bright structures, rough surfaces, speckle, layover, or shadow (Xiao et al., 10 Apr 2026).

The benchmark also exposes disciplinary gaps in GIS and photogrammetry. Map recognition, layer interpretation, camera geometry, GSD, and parallax remain difficult even for top models. These weaknesses are consistent with the broader geospatial VLM literature: GEOBench-VLM reports that existing VLMs face persistent challenges in geospatial-specific tasks such as temporal analysis, large-scale object counting, tiny object detection, and spatial relation understanding, with the best reported MCQ performance still only slightly above double random guessing (Danish et al., 2024).

A practical implication is that GeoMMBench is not merely testing whether a model can recognize objects in remote-sensing imagery. It is testing whether the model can combine technical visual parsing, geoscientific priors, and multi-step reasoning under modality shift. The paper’s failure analysis suggests that current models often possess fragments of the required textual knowledge but cannot reliably bind that knowledge to geospatial evidence (Xiao et al., 10 Apr 2026).

5. GeoMMAgent and tool-augmented execution

GeoMMAgent is the accompanying multi-agent framework proposed to address the weaknesses surfaced by GeoMMBench. Its architecture is organized into three stages: Planning, Multi-Agent Execution, and Self-Evaluation. A Coordinate Agent interprets the image-query pair and produces an execution plan; specialized agents then perform perception, retrieval, and reasoning; finally, a Self-Evaluation Agent inspects the reasoning trace and answer, and can trigger re-execution if confidence is low or the evidence is insufficient (Xiao et al., 10 Apr 2026).

The framework exposes tools through Model Context Protocol endpoints and groups them into four toolkits. The General Toolkit handles image preprocessing and measurement operations such as cropping, scaling, super-resolution, area counting, and box counting. The Knowledge Toolkit includes Google API, Wikimedia API, and GME for multimodal retrieval. The Perception Toolkit includes a YOLOv11 scene classification model trained on Million-AID, a YOLOv11 detection model trained on DOTA-v2, and a DeepLabv3+ segmentation model trained on LoveDA. The Reasoning Toolkit uses Qwen-VL-Max as the reasoning agent (Xiao et al., 10 Apr 2026).

The paper provides two concrete execution patterns. In one case, the system is asked to identify the meaning of a marked electromagnetic band; initial reasoning selects an incomplete option, self-evaluation detects missing justification, retrieval is refined toward microwave-specific evidence, and the final answer is corrected. In another case, the task is aircraft counting; the coordinator dispatches to the detection toolkit, the detector returns plane boxes, the box-counting tool outputs 12, and the reasoning agent maps the count to the correct option. These examples are used to show that monolithic MLLMs and tool-augmented agents fail for different reasons: the former often fail in grounded perception or retrieval, whereas the latter can decompose the task into specialized operations (Xiao et al., 10 Apr 2026).

GeoMMAgent attains 86.5% on the validation set and 88.4% on the test set, exceeding both the strongest standalone MLLMs and the measured human validation score. Its test accuracy by discipline is 87.6% for RS, 89.8% for photogrammetry, 93.2% for GIS, and 97.6% for GNSS. By modality it reaches 78.2% on optical, 94.0% on DEM, 91.2% on SAR, 89.2% on HSI, 74.2% on LiDAR, and 89.3% on thermal. By task type it reaches 92.2% on principles, 82.9% on perception, 97.3% on spatial, 76.9% on quality, 97.8% on time-series, and 83.1% on applications (Xiao et al., 10 Apr 2026).

The ablation study shows how the performance gain is distributed across modules.

Variant	Val	Test
Full GeoMMAgent	86.5%	88.4%
w/o Knowledge	83.8%	87.4%
w/o Perception	83.8%	80.3%
w/o Reasoning	59.5%	67.3%
w/o Self-evaluation	81.1%	80.1%

The reasoning component produces the largest drop when removed, while self-evaluation also yields a large gain. The paper interprets this as evidence that expert-level geospatial performance is not obtained by a stronger base model alone, but by strategic integration of retrieval, perception, reasoning, and iterative verification (Xiao et al., 10 Apr 2026).

6. Research context, distinction from neighboring benchmarks, and limitations

GeoMMBench sits in a broader benchmark landscape but occupies a specific niche. The paper characterizes earlier remote-sensing VQA and vision-language benchmarks such as EarthVQA, LHRS-Bench, GeoChat-Bench, VRSBench, TeoChat-Bench, XLRS-Bench, and CHOICE as narrow in domains, task-limited, and single-modality dominant. GEOBench-VLM broadens geospatial evaluation to scene understanding, object counting, localization, fine-grained categorization, segmentation, temporal analysis, and non-optical understanding over more than 10,000 manually verified instructions, while GEO-Bench-2 systematizes capability-based evaluation for geospatial foundation models across 19 datasets and tasks such as classification, segmentation, regression, detection, and instance segmentation (Danish et al., 2024, Simumba et al., 19 Nov 2025).

GeoMMBench differs from both. Relative to GEOBench-VLM, it adds explicit coverage of photogrammetry, GIS, and GNSS, and it foregrounds expert-written multiple-choice QA over technical visual forms such as maps, plots, diagrams, tables, and mathematical notation. Relative to GEO-Bench-2, it is a vision-language benchmark rather than a benchmark for geospatial foundation models without a language interface (Xiao et al., 10 Apr 2026).

A common source of confusion is the similarity of names to geometry benchmarks. GeoEval, GeoSense, SolidGeo, and GeoBench evaluate multimodal geometric problem solving, theorem application, solid geometry, or hierarchical reasoning over diagrams; GeoMMBench instead addresses geoscience and remote sensing, with emphasis on remote sensing, photogrammetry, GIS, GNSS, and heterogeneous geospatial sensing (Zhang et al., 2024, Xu et al., 17 Apr 2025, Wang et al., 27 May 2025, Feng et al., 30 Dec 2025).

The paper also states several limitations. GeoMMBench’s 1,053 questions cannot cover all geoscience and remote-sensing knowledge, and manual curation may introduce selection bias. GeoMMAgent’s tool library is tuned toward the tasks present in GeoMMBench and does not constitute a complete remote-sensing toolbox. Accordingly, strong performance on GeoMMBench is presented as necessary but not sufficient for expert AGI in geoscience and remote sensing. At the same time, the benchmark and agent are described as training-free and extensible, with Model Context Protocol support intended to permit the addition of new tools, new perception modules, and new retrieval sources without changing the overall orchestration design (Xiao et al., 10 Apr 2026).