---
title: 'MapIQ: Multimodal Map Reading Benchmark'
url: https://www.emergentmind.com/topics/mapiq
type: topic
---

# MapIQ: Multimodal Map Reading Benchmark

Searching arXiv for MapIQ and closely related benchmark papers.
MapIQ is a benchmark dataset and experimental framework for evaluating multimodal large language models (MLLMs) on low-level map-reading tasks, introduced to address limitations in prior map question answering research that focused primarily on choropleth maps, a narrow task set, and limited thematic coverage [2507.11625]. It comprises a curated collection of thematic maps of the USA and 14,706 question–answer pairs spanning choropleth maps, cartograms, and proportional symbol maps, six thematic domains, six visual analytical tasks, and multiple question formats, and it is used to benchmark seven MLLMs against a human baseline and under 15 map design variations [2507.11625]. In related benchmark ecosystems, MapIQ has also been incorporated into evidence-grounded diagram reasoning evaluation, where it contributes map-based question instances requiring localization of the visual evidence needed to justify answers [2604.25231].

## 1. Scope, motivation, and benchmark definition

Map question answering, or Map-VQA, asks a model to answer questions about a map image: identify where values are high or low, compare regions, detect clusters, and related operations. The problem is motivated by real-world applications in policy analysis, public health, environmental monitoring, and crime analysis, and by the fact that maps require reasoning over geospatial structure, legends, projections, spatial relations, and thematic context rather than generic object recognition alone [2507.11625].

MapIQ was introduced to fill three stated gaps in prior Map-VQA work. First, prior benchmarks such as MapQA and MAPWise focused almost exclusively on choropleth maps, whereas cartograms and proportional symbol maps had not been systematically studied [2507.11625]. Second, earlier work primarily emphasized basic literacy, value lookup, and finding extremes, rather than a broader set of low-level visual analytics tasks catalogued by visualization and GIS literature [2507.11625]. Third, existing datasets often used a single theme, leaving it unclear how model performance varies across domains such as housing, crime, and environment or whether models exhibit topic-specific biases [2507.11625].

Within this framing, MapIQ is defined not merely as a static corpus but as a benchmark and experimental framework. Its stated goals are to measure how well MLLMs can read, interpret, and reason about thematic maps at local and global scales; compare performance across map types, tasks, question formats, and themes; investigate robustness to changes in map design; and contrast open-source and closed-source models with human experts [2507.11625]. A plausible implication is that MapIQ treats map understanding as a controlled multimodal reasoning problem rather than as a narrow OCR or legend-parsing task.

## 2. Dataset composition and cartographic design

MapIQ contains 774 unique map images generated from 258 underlying datasets and rendered in three map types: choropleth maps, cartograms, and proportional symbol maps [2507.11625]. Choropleth maps use five discrete classes via Fisher–Jenks classification and a sequential blue ColorBrewer scheme; cartograms are hexbin maps with one hexagonal cell per state and the same 5-class sequential color encoding; proportional symbol maps show states with circles whose size encodes the data value [2507.11625]. All maps use the Albers USA projection on a white background, include two-letter state abbreviations with leader lines if needed, and apply normalization where appropriate [2507.11625].

The benchmark covers six themes commonly used in thematic cartography: Economic, Housing, Social, Health, Crime, and Environment [2507.11625]. The underlying metadata are sourced from U.S. Census ACS, EPA, weather.gov, FBI Crime Data Explorer, CDC, and related sources, and the geographic coverage is US states, with analysis zones defined as West, Midwest, Northeast, and South [2507.11625]. For experiments, the test set was sampled so that question instances are balanced across themes even though the raw dataset is not [2507.11625].

| Component | Value | Notes |
|---|---:|---|
| Underlying datasets | 258 | Distinct topics |
| Map images | 774 | \(258 \times 3\) map types |
| QA pairs | 14,706 | 19 questions per map |

The dataset statistics are explicitly defined as \(774\ \text{maps} \times 19\ \text{questions per map} = 14{,}706\ \text{QA pairs}\) [2507.11625]. Question formats include binary, multiple-choice, single-value, and list, and the experimental test subset contains 5,130 QA pairs, approximately 35% of the full set, stratified by map type, task type, question type, and theme [2507.11625].

The choice of cartographic forms is methodologically important. Choropleths and cartograms provide color-based encodings, whereas proportional symbol maps provide size-based encodings, enabling direct study of whether MLLMs exhibit a bias toward the more common choropleth format [2507.11625]. This suggests that MapIQ is designed to separate failures of spatial reasoning from failures of visual encoding interpretation.

## 3. Task taxonomy, question generation, and ground truth

MapIQ is built on six visual analytical tasks covering local and global reasoning [2507.11625]. These are Retrieve Value, Pairwise Point Comparisons, Spatial Extremes, Spatial Clusters, Determine Range, and Regional Comparisons [2507.11625]. Retrieve Value tests legend reading for a specific state; Pairwise Point Comparisons require relational comparison between two states; Spatial Extremes identify highest or lowest class values locally or globally; Spatial Clusters detect contiguous states sharing Class 1 or Class 5 values under a queen adjacency rule and with cluster size at least three states; Determine Range requires identifying the minimum and maximum class present within a region; and Regional Comparisons require summarizing aggregate patterns between zones [2507.11625].

The question system uses 19 manually designed templates distributed across the supported answer formats [2507.11625]. Binary and multiple-choice questions are used for all six tasks; single-value questions are used only for Retrieve Value and Determine Range; list questions are used for all tasks except Determine Range [2507.11625]. Template-based generation fills placeholders such as `[state name]`, `[USA/map zone]`, `[class]`, and `[range]`, thereby producing systematic local and global variants [2507.11625].

A central design feature is that all answers are extracted programmatically from the underlying GeoJSON and metadata rather than by manual inspection of images [2507.11625]. For each task–question combination, scripts compute the correct states, classes, or ranges, and an expert reviews samples and scripts iteratively to ensure accuracy [2507.11625]. This sharply distinguishes MapIQ from manually judged visual QA settings in which annotation errors may enter through subjective image reading.

The task design also formalizes spatial reasoning primitives that are often implicit in map reading. The spatial cluster task explicitly defines a cluster as geographically proximate locations sharing the same attribute class under the queen adjacency rule, limited to Class 1 and Class 5, with minimum cluster size at least three states [2507.11625]. In later diagram-grounding work, MapIQ examples were used precisely because they require evidence not only from answer-bearing states but also from comparison states and legends, such as in a hexagonal cartogram question about the Northeast zone relative to New York [2604.25231]. A plausible implication is that MapIQ’s question design makes it suitable both for answer accuracy evaluation and for evidence-grounding analysis.

## 4. Evaluation protocol, models, and human baseline

The benchmark evaluates three closed-source models—ChatGPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—and four open-source models at the 7B–8B scale—Qwen2-VL, Molmo, InternVL2.5-MPO, and Idefics3 [2507.11625]. Two additional models, DeepSeek-VL2 and MiniCPM-V 2.6, were tested but excluded due to incoherent or missing outputs [2507.11625]. All evaluations are zero-shot, with no fine-tuning on MapIQ and no in-context training examples [2507.11625].

Each input consists of the map image and a task-specific text prompt [2507.11625]. Prompts include general instructions, rigid answer-format requirements, and, where necessary, zoning information listing the four US zones and their member states [2507.11625]. For Spatial Clusters, prompts explicitly define the cluster rules, class restrictions, queen adjacency, and minimum cluster size [2507.11625]. Humans receive the same instructions [2507.11625].

Evaluation is metric-specific. Binary and single-value questions are scored by accuracy, while MCQ and list questions are scored by per-response F1 and then averaged [2507.11625]. Outputs are parsed and cleaned manually to extract answer tokens, normalize malformed or inconsistent formats, and handle invalid combinations such as selecting “None of the above” plus other choices [2507.11625]. This protocol reflects the fact that response formatting, rather than only reasoning, is a nontrivial confound in multimodal LLM evaluation.

The human baseline is collected on a subset of 450 QA pairs, about 9% of the test set, balanced across map types, task types, question types, and themes [2507.11625]. Two expert human map readers independently answer all questions, and a third expert validates responses for consistency [2507.11625]. The use of expert map readers rather than crowd workers places the comparison against a technically informed reading baseline.

## 5. Empirical findings

Claude 3.5 Sonnet is reported as the best-performing model overall and is top across task type, map type, theme, and question type in most conditions [2507.11625]. It is particularly strong on Regional Comparisons, with 65.84% accuracy [2507.11625]. Qwen2-VL is the strongest open-source model, often second-best overall, and is best on Retrieve Value with 61.89% accuracy and on Determine Range among all models, while also reaching 75.25% accuracy on binary questions [2507.11625]. Molmo and Idefics3 are consistently weakest, with Idefics3 reaching 15.56% on Determine Range [2507.11625].

By task type, the reported order from easiest to hardest for MLLMs is Regional Comparisons, Spatial Extremes, Pairwise Point Comparisons, Retrieve Value, Spatial Clusters, and Determine Range [2507.11625]. One of the most notable findings is that Retrieve Value, which is simple for humans, is surprisingly difficult for MLLMs, whereas Regional Comparisons, although conceptually more global, are relatively easier [2507.11625]. The paper explicitly suggests that this may reflect reliance on approximate pattern recognition and world knowledge in regional judgments rather than robust pixel-to-legend grounding [2507.11625].

By map type, the general ordering is choropleth first, cartogram second, and proportional symbol last [2507.11625]. Claude 3.5 Sonnet is best on all three types, though some exceptions appear: Claude and Molmo show slightly better accuracy on cartograms than on choropleths, and Gemini 1.5 Pro performs best on proportional symbol maps [2507.11625]. The overall trend is interpreted as suggesting a bias toward choropleth maps, likely reflecting their prevalence in training data [2507.11625].

By theme, Environment is easiest and Crime is hardest, with accuracy ranging from about 47.16% down to 37.55% [2507.11625]. Gemini 1.5 Pro is particularly weak on Crime at 21.85%, the lowest model–theme combination reported [2507.11625]. By question type, binary is easiest and single value is hardest, while list questions slightly outperform MCQs for MLLMs even though MCQs are easier for humans [2507.11625]. This suggests that models may often identify multiple relevant items but struggle to choose among highly similar predefined options.

Closed-source models outperform open-source models by roughly 4.4% on average, with the gap especially small for choropleths at 0.08% and largest for proportional symbol maps at about 6.29% [2507.11625]. Determine Range is an exception where open-source models do better, mainly because ChatGPT-4o is weak at 16.30% [2507.11625]. Yet the dominant comparison is against human experts: humans outperform MLLMs by approximately 50.35 percentage points across experimental variables, with the largest gap on Determine Range at about 68.79% and on proportional symbol maps at about 52.48% [2507.11625]. The paper therefore concludes that MLLMs remain far from human-level map-reading proficiency on low-level tasks [2507.11625].

## 6. Robustness, grounding, and position within related work

MapIQ includes a map design variation experiment with 15 variations grouped into five categories: label size, legend properties, color schemes, map orientation, and removal of key elements such as title or legend [2507.11625]. The variation subset uses 36 maps, rendered into 540 varied maps with 684 associated QA pairs, and tests Claude 3.5 Sonnet and Qwen2-VL relative to baseline performance [2507.11625]. Qwen2-VL shows an average performance change of \(-1.68\%\) across task types and \(-2.96\%\) across map types, with all 15 variations reducing accuracy across map types [2507.11625]. Claude 3.5 Sonnet is more robust overall, with average changes of \(-0.65\%\) across tasks and \(-1.45\%\) across map types, and only 7 of 15 variations reducing performance [2507.11625].

For both models, the Color Flipped variation—where lighter shades indicate higher classes—causes the strongest performance drop [2507.11625]. Qwen2-VL is especially harmed by No legend on proportional symbol maps, with a \(-9.09\%\) drop, while Claude improves by \(+6.98\%\) under No legend for choropleths, suggesting increased reliance on relative spatial patterns and internal heuristics when the legend is absent [2507.11625]. The broader interpretation offered is that current MLLMs are sensitive to canonical color semantics, symbol-size interpretation, and title-provided thematic context [2507.11625].

MapIQ also has significance beyond its original benchmark. In DRAGON, a benchmark for evidence-grounded visual reasoning over diagrams, MapIQ contributes 369 QA instances and 182 images in the released test set [2604.25231]. There, evidence annotations for MapIQ questions may include state or region cells, labels, legends, and comparison regions, and performance patterns show that even when models can roughly localize relevant states and legends, they often fail to cover the full evidence set needed for justified reasoning [2604.25231]. This suggests that MapIQ probes not only answer correctness but also the extent to which answers are grounded in the diagram itself.

Relative to prior map benchmarks, MAPWise evaluates advanced queries over choropleth maps from the United States, India, and China with 43 templates, but remains choropleth-only [2409.00255]. MapIQ extends the benchmark space by combining multiple map types, six well-defined low-level visual analytical tasks, six themes and 258 real datasets, systematic design-variation testing, and comparisons across open and closed models and human baselines [2507.11625]. In this sense, MapIQ occupies a distinct position at the intersection of cartography, visual analytics, and multimodal LLM evaluation.

Source: https://www.emergentmind.com/topics/mapiq