MapIQ: Multimodal Map Reading Benchmark
- MapIQ is a benchmark dataset and experimental framework for evaluating multimodal LLMs on thematic maps through low-level visual analytical tasks.
- It comprises 14,706 QA pairs generated from 774 map images in choropleth, cartogram, and proportional symbol formats covering six thematic domains.
- The framework assesses spatial reasoning with systematic map design variations and compares model performance against expert human baseline.
Searching arXiv for MapIQ and closely related benchmark papers. MapIQ is a benchmark dataset and experimental framework for evaluating multimodal LLMs (MLLMs) on low-level map-reading tasks, introduced to address limitations in prior map question answering research that focused primarily on choropleth maps, a narrow task set, and limited thematic coverage (Srivastava et al., 15 Jul 2025). It comprises a curated collection of thematic maps of the USA and 14,706 question–answer pairs spanning choropleth maps, cartograms, and proportional symbol maps, six thematic domains, six visual analytical tasks, and multiple question formats, and it is used to benchmark seven MLLMs against a human baseline and under 15 map design variations (Srivastava et al., 15 Jul 2025). In related benchmark ecosystems, MapIQ has also been incorporated into evidence-grounded diagram reasoning evaluation, where it contributes map-based question instances requiring localization of the visual evidence needed to justify answers (Iyengar et al., 28 Apr 2026).
1. Scope, motivation, and benchmark definition
Map question answering, or Map-VQA, asks a model to answer questions about a map image: identify where values are high or low, compare regions, detect clusters, and related operations. The problem is motivated by real-world applications in policy analysis, public health, environmental monitoring, and crime analysis, and by the fact that maps require reasoning over geospatial structure, legends, projections, spatial relations, and thematic context rather than generic object recognition alone (Srivastava et al., 15 Jul 2025).
MapIQ was introduced to fill three stated gaps in prior Map-VQA work. First, prior benchmarks such as MapQA and MAPWise focused almost exclusively on choropleth maps, whereas cartograms and proportional symbol maps had not been systematically studied (Srivastava et al., 15 Jul 2025). Second, earlier work primarily emphasized basic literacy, value lookup, and finding extremes, rather than a broader set of low-level visual analytics tasks catalogued by visualization and GIS literature (Srivastava et al., 15 Jul 2025). Third, existing datasets often used a single theme, leaving it unclear how model performance varies across domains such as housing, crime, and environment or whether models exhibit topic-specific biases (Srivastava et al., 15 Jul 2025).
Within this framing, MapIQ is defined not merely as a static corpus but as a benchmark and experimental framework. Its stated goals are to measure how well MLLMs can read, interpret, and reason about thematic maps at local and global scales; compare performance across map types, tasks, question formats, and themes; investigate robustness to changes in map design; and contrast open-source and closed-source models with human experts (Srivastava et al., 15 Jul 2025). A plausible implication is that MapIQ treats map understanding as a controlled multimodal reasoning problem rather than as a narrow OCR or legend-parsing task.
2. Dataset composition and cartographic design
MapIQ contains 774 unique map images generated from 258 underlying datasets and rendered in three map types: choropleth maps, cartograms, and proportional symbol maps (Srivastava et al., 15 Jul 2025). Choropleth maps use five discrete classes via Fisher–Jenks classification and a sequential blue ColorBrewer scheme; cartograms are hexbin maps with one hexagonal cell per state and the same 5-class sequential color encoding; proportional symbol maps show states with circles whose size encodes the data value (Srivastava et al., 15 Jul 2025). All maps use the Albers USA projection on a white background, include two-letter state abbreviations with leader lines if needed, and apply normalization where appropriate (Srivastava et al., 15 Jul 2025).
The benchmark covers six themes commonly used in thematic cartography: Economic, Housing, Social, Health, Crime, and Environment (Srivastava et al., 15 Jul 2025). The underlying metadata are sourced from U.S. Census ACS, EPA, weather.gov, FBI Crime Data Explorer, CDC, and related sources, and the geographic coverage is US states, with analysis zones defined as West, Midwest, Northeast, and South (Srivastava et al., 15 Jul 2025). For experiments, the test set was sampled so that question instances are balanced across themes even though the raw dataset is not (Srivastava et al., 15 Jul 2025).
| Component | Value | Notes |
|---|---|---|
| Underlying datasets | 258 | Distinct topics |
| Map images | 774 | map types |
| QA pairs | 14,706 | 19 questions per map |
The dataset statistics are explicitly defined as (Srivastava et al., 15 Jul 2025). Question formats include binary, multiple-choice, single-value, and list, and the experimental test subset contains 5,130 QA pairs, approximately 35% of the full set, stratified by map type, task type, question type, and theme (Srivastava et al., 15 Jul 2025).
The choice of cartographic forms is methodologically important. Choropleths and cartograms provide color-based encodings, whereas proportional symbol maps provide size-based encodings, enabling direct study of whether MLLMs exhibit a bias toward the more common choropleth format (Srivastava et al., 15 Jul 2025). This suggests that MapIQ is designed to separate failures of spatial reasoning from failures of visual encoding interpretation.
3. Task taxonomy, question generation, and ground truth
MapIQ is built on six visual analytical tasks covering local and global reasoning (Srivastava et al., 15 Jul 2025). These are Retrieve Value, Pairwise Point Comparisons, Spatial Extremes, Spatial Clusters, Determine Range, and Regional Comparisons (Srivastava et al., 15 Jul 2025). Retrieve Value tests legend reading for a specific state; Pairwise Point Comparisons require relational comparison between two states; Spatial Extremes identify highest or lowest class values locally or globally; Spatial Clusters detect contiguous states sharing Class 1 or Class 5 values under a queen adjacency rule and with cluster size at least three states; Determine Range requires identifying the minimum and maximum class present within a region; and Regional Comparisons require summarizing aggregate patterns between zones (Srivastava et al., 15 Jul 2025).
The question system uses 19 manually designed templates distributed across the supported answer formats (Srivastava et al., 15 Jul 2025). Binary and multiple-choice questions are used for all six tasks; single-value questions are used only for Retrieve Value and Determine Range; list questions are used for all tasks except Determine Range (Srivastava et al., 15 Jul 2025). Template-based generation fills placeholders such as [state name], [USA/map zone], [class], and [range], thereby producing systematic local and global variants (Srivastava et al., 15 Jul 2025).
A central design feature is that all answers are extracted programmatically from the underlying GeoJSON and metadata rather than by manual inspection of images (Srivastava et al., 15 Jul 2025). For each task–question combination, scripts compute the correct states, classes, or ranges, and an expert reviews samples and scripts iteratively to ensure accuracy (Srivastava et al., 15 Jul 2025). This sharply distinguishes MapIQ from manually judged visual QA settings in which annotation errors may enter through subjective image reading.
The task design also formalizes spatial reasoning primitives that are often implicit in map reading. The spatial cluster task explicitly defines a cluster as geographically proximate locations sharing the same attribute class under the queen adjacency rule, limited to Class 1 and Class 5, with minimum cluster size at least three states (Srivastava et al., 15 Jul 2025). In later diagram-grounding work, MapIQ examples were used precisely because they require evidence not only from answer-bearing states but also from comparison states and legends, such as in a hexagonal cartogram question about the Northeast zone relative to New York (Iyengar et al., 28 Apr 2026). A plausible implication is that MapIQ’s question design makes it suitable both for answer accuracy evaluation and for evidence-grounding analysis.
4. Evaluation protocol, models, and human baseline
The benchmark evaluates three closed-source models—ChatGPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—and four open-source models at the 7B–8B scale—Qwen2-VL, Molmo, InternVL2.5-MPO, and Idefics3 (Srivastava et al., 15 Jul 2025). Two additional models, DeepSeek-VL2 and MiniCPM-V 2.6, were tested but excluded due to incoherent or missing outputs (Srivastava et al., 15 Jul 2025). All evaluations are zero-shot, with no fine-tuning on MapIQ and no in-context training examples (Srivastava et al., 15 Jul 2025).
Each input consists of the map image and a task-specific text prompt (Srivastava et al., 15 Jul 2025). Prompts include general instructions, rigid answer-format requirements, and, where necessary, zoning information listing the four US zones and their member states (Srivastava et al., 15 Jul 2025). For Spatial Clusters, prompts explicitly define the cluster rules, class restrictions, queen adjacency, and minimum cluster size (Srivastava et al., 15 Jul 2025). Humans receive the same instructions (Srivastava et al., 15 Jul 2025).
Evaluation is metric-specific. Binary and single-value questions are scored by accuracy, while MCQ and list questions are scored by per-response F1 and then averaged (Srivastava et al., 15 Jul 2025). Outputs are parsed and cleaned manually to extract answer tokens, normalize malformed or inconsistent formats, and handle invalid combinations such as selecting “None of the above” plus other choices (Srivastava et al., 15 Jul 2025). This protocol reflects the fact that response formatting, rather than only reasoning, is a nontrivial confound in multimodal LLM evaluation.
The human baseline is collected on a subset of 450 QA pairs, about 9% of the test set, balanced across map types, task types, question types, and themes (Srivastava et al., 15 Jul 2025). Two expert human map readers independently answer all questions, and a third expert validates responses for consistency (Srivastava et al., 15 Jul 2025). The use of expert map readers rather than crowd workers places the comparison against a technically informed reading baseline.
5. Empirical findings
Claude 3.5 Sonnet is reported as the best-performing model overall and is top across task type, map type, theme, and question type in most conditions (Srivastava et al., 15 Jul 2025). It is particularly strong on Regional Comparisons, with 65.84% accuracy (Srivastava et al., 15 Jul 2025). Qwen2-VL is the strongest open-source model, often second-best overall, and is best on Retrieve Value with 61.89% accuracy and on Determine Range among all models, while also reaching 75.25% accuracy on binary questions (Srivastava et al., 15 Jul 2025). Molmo and Idefics3 are consistently weakest, with Idefics3 reaching 15.56% on Determine Range (Srivastava et al., 15 Jul 2025).
By task type, the reported order from easiest to hardest for MLLMs is Regional Comparisons, Spatial Extremes, Pairwise Point Comparisons, Retrieve Value, Spatial Clusters, and Determine Range (Srivastava et al., 15 Jul 2025). One of the most notable findings is that Retrieve Value, which is simple for humans, is surprisingly difficult for MLLMs, whereas Regional Comparisons, although conceptually more global, are relatively easier (Srivastava et al., 15 Jul 2025). The paper explicitly suggests that this may reflect reliance on approximate pattern recognition and world knowledge in regional judgments rather than robust pixel-to-legend grounding (Srivastava et al., 15 Jul 2025).
By map type, the general ordering is choropleth first, cartogram second, and proportional symbol last (Srivastava et al., 15 Jul 2025). Claude 3.5 Sonnet is best on all three types, though some exceptions appear: Claude and Molmo show slightly better accuracy on cartograms than on choropleths, and Gemini 1.5 Pro performs best on proportional symbol maps (Srivastava et al., 15 Jul 2025). The overall trend is interpreted as suggesting a bias toward choropleth maps, likely reflecting their prevalence in training data (Srivastava et al., 15 Jul 2025).
By theme, Environment is easiest and Crime is hardest, with accuracy ranging from about 47.16% down to 37.55% (Srivastava et al., 15 Jul 2025). Gemini 1.5 Pro is particularly weak on Crime at 21.85%, the lowest model–theme combination reported (Srivastava et al., 15 Jul 2025). By question type, binary is easiest and single value is hardest, while list questions slightly outperform MCQs for MLLMs even though MCQs are easier for humans (Srivastava et al., 15 Jul 2025). This suggests that models may often identify multiple relevant items but struggle to choose among highly similar predefined options.
Closed-source models outperform open-source models by roughly 4.4% on average, with the gap especially small for choropleths at 0.08% and largest for proportional symbol maps at about 6.29% (Srivastava et al., 15 Jul 2025). Determine Range is an exception where open-source models do better, mainly because ChatGPT-4o is weak at 16.30% (Srivastava et al., 15 Jul 2025). Yet the dominant comparison is against human experts: humans outperform MLLMs by approximately 50.35 percentage points across experimental variables, with the largest gap on Determine Range at about 68.79% and on proportional symbol maps at about 52.48% (Srivastava et al., 15 Jul 2025). The paper therefore concludes that MLLMs remain far from human-level map-reading proficiency on low-level tasks (Srivastava et al., 15 Jul 2025).
6. Robustness, grounding, and position within related work
MapIQ includes a map design variation experiment with 15 variations grouped into five categories: label size, legend properties, color schemes, map orientation, and removal of key elements such as title or legend (Srivastava et al., 15 Jul 2025). The variation subset uses 36 maps, rendered into 540 varied maps with 684 associated QA pairs, and tests Claude 3.5 Sonnet and Qwen2-VL relative to baseline performance (Srivastava et al., 15 Jul 2025). Qwen2-VL shows an average performance change of across task types and across map types, with all 15 variations reducing accuracy across map types (Srivastava et al., 15 Jul 2025). Claude 3.5 Sonnet is more robust overall, with average changes of across tasks and across map types, and only 7 of 15 variations reducing performance (Srivastava et al., 15 Jul 2025).
For both models, the Color Flipped variation—where lighter shades indicate higher classes—causes the strongest performance drop (Srivastava et al., 15 Jul 2025). Qwen2-VL is especially harmed by No legend on proportional symbol maps, with a drop, while Claude improves by under No legend for choropleths, suggesting increased reliance on relative spatial patterns and internal heuristics when the legend is absent (Srivastava et al., 15 Jul 2025). The broader interpretation offered is that current MLLMs are sensitive to canonical color semantics, symbol-size interpretation, and title-provided thematic context (Srivastava et al., 15 Jul 2025).
MapIQ also has significance beyond its original benchmark. In DRAGON, a benchmark for evidence-grounded visual reasoning over diagrams, MapIQ contributes 369 QA instances and 182 images in the released test set (Iyengar et al., 28 Apr 2026). There, evidence annotations for MapIQ questions may include state or region cells, labels, legends, and comparison regions, and performance patterns show that even when models can roughly localize relevant states and legends, they often fail to cover the full evidence set needed for justified reasoning (Iyengar et al., 28 Apr 2026). This suggests that MapIQ probes not only answer correctness but also the extent to which answers are grounded in the diagram itself.
Relative to prior map benchmarks, MAPWise evaluates advanced queries over choropleth maps from the United States, India, and China with 43 templates, but remains choropleth-only (Mukhopadhyay et al., 2024). MapIQ extends the benchmark space by combining multiple map types, six well-defined low-level visual analytical tasks, six themes and 258 real datasets, systematic design-variation testing, and comparisons across open and closed models and human baselines (Srivastava et al., 15 Jul 2025). In this sense, MapIQ occupies a distinct position at the intersection of cartography, visual analytics, and multimodal LLM evaluation.
