---
title: 'CartoMapQA: Benchmark for Cartographic Map QA'
url: https://www.emergentmind.com/topics/cartomapqa
type: topic
---

# CartoMapQA: Benchmark for Cartographic Map QA

Searching arXiv for the CartoMapQA paper and closely related map-QA benchmarks to ground the article in current literature.
CartoMapQA is a benchmark dataset and evaluation suite for testing how well large vision-language models understand cartographic maps through question-answering tasks. It is organized as a hierarchically structured map-understanding benchmark spanning low-, mid-, and high-level skills, and each sample contains either a cartographic map image or an isolated map-feature image together with a question or request and a ground-truth answer. The dataset contains **2,251 questions** over **853 OpenStreetMap-derived maps**, covers **six tasks** grouped under **map information understanding**, **scale and distance interpretation**, and **directional reasoning / navigation**, and was introduced to expose persistent weaknesses of contemporary vision-language models in map-specific semantics, OCR on map text, and geospatial reasoning [2512.03558].

## 1. Definition and scope

CartoMapQA is defined as a question-answering benchmark for cartographic maps rather than a generic visual question answering resource. Its samples combine a map image or isolated map-feature image with a question and a ground-truth answer, and its task design explicitly targets map-specific operations such as symbol recognition, embedded information extraction, scale interpretation, marker localization, and route-based reasoning [2512.03558].

The benchmark is motivated by the claim that cartographic maps occupy a distinctive position among geospatial representations. In the benchmark description, text-only inputs are described as limited by tokenization and often unable to preserve rich spatial structure, while satellite imagery is described as visually rich but cluttered with irrelevant detail. Cartographic maps, by contrast, provide a structured and semantically abstracted representation through standardized symbols, color-coded areas, roads and route structure, textual labels, and scale bars. This framing is important because it makes CartoMapQA a benchmark for cartographic interpretation as such, not merely for generic OCR or image captioning [2512.03558].

The paper presents CartoMapQA as a diagnostic benchmark rather than a saturated leaderboard. Its central claim is that current multimodal models can appear strong on general multimodal reasoning while still failing at map-specific operations. A plausible implication is that map understanding should be evaluated as a hierarchy of dependent competencies rather than as a single end-to-end score.

## 2. Dataset composition and task hierarchy

CartoMapQA contains **2,251** questions distributed across **six tasks**: **MFSEM**, **STMF**, **MTMF**, **RLEST**, **MMLOC**, and **SRNAV**. These tasks are organized to separate foundational perceptual competence from higher-order geospatial reasoning, so that failures in navigation can be traced back to weaker capabilities such as symbol interpretation, label extraction, or marker grounding [2512.03558].

| Task | Questions | Core operation |
|---|---:|---|
| MFSEM | 463 | Isolated map-feature semantics |
| STMF | 510 | Single-type feature presence, counting, name listing |
| MTMF | 150 | Multiple-type feature counting and name listing |
| RLEST | 600 | Route length estimation with scale bars |
| MMLOC | 250 | Marker localization by road-name intersection |
| SRNAV | 278 | Shortest-route instruction generation |

The hierarchy is explicit. **MFSEM** tests isolated feature semantics without broader map context. **STMF** and **MTMF** move to full-map feature analysis. **MMLOC** serves as a bridge from map reading to route reasoning by requiring a marker to be grounded to two intersecting road names. **RLEST** tests the use of scale bars for quantitative estimation. **SRNAV** is the highest-level task and asks for a shortest drivable route in a structured turn-by-turn format [2512.03558].

The answer formats are deliberately mixed. **MFSEM** is multiple-choice, while the remaining tasks are predominantly open-ended. Across the whole benchmark, **463 questions (21%)** are multiple-choice and **1,788 (79%)** are open-ended. This matters because the benchmark does not collapse map QA into a single response regime; instead it spans classification, binary decisions, counting, name listing, numeric estimation, and structured route generation [2512.03558].

The benchmark also specifies image and map distributions. Rendered maps have size **1366 × 768 pixels**. The maps use zoom levels **15, 16, 17, 18, 19**, with counts **1, 7, 213, 187, 445** respectively. The data come from **San Jose**, **Manchester**, and **Melbourne**, chosen from **three English-speaking countries** in order to reduce confounding from multilingual text comprehension [2512.03558].

## 3. Construction methodology and ground-truth generation

CartoMapQA is built from **OpenStreetMap** rather than proprietary services, and the paper explicitly cites licensing openness and reproducibility as reasons for this design choice. Map rendering uses **Folium**, while graph-based path computation and related operations use **OSMnx** and **NetworkX**. This makes the benchmark technically reproducible and aligns it with open data workflows [2512.03558].

The dataset construction pipeline differs by task. **MFSEM** uses isolated OpenStreetMap feature images. Its distractor choices are created through a combination of **CLIP** image embeddings and semantic class structure: two most similar features from the same class, one most similar feature from a different class, and one randomly selected feature, together with the correct label, yielding five options whose order is randomly shuffled. This design makes the multiple-choice setting nontrivial by avoiding obviously unrelated distractors [2512.03558].

For **STMF** and **MTMF**, the authors render full cartographic maps from OSM, extract map-feature data within each map boundary, and then **manually verify** counts and names because rendered content may diverge from underlying OSM extraction due to rendering limitations, outdated entries, or label mismatches. For **RLEST**, **MMLOC**, and **SRNAV**, ground truth is derived with **OSMnx + NetworkX** and then manually checked where needed. The paper states that routes in **SRNAV** were manually verified and corrected for consistency with rendered maps, and that **300** initial routes were reduced to **278 valid routes** because some were filtered out for readability issues such as overlapping map elements [2512.03558].

The benchmark’s tasks encode several exact conventions. In **MMLOC**, the answer is the pair of road names intersecting at a colored marker. In **RLEST**, maps include two blue markers, a predefined blue path, and a scale bar shown in **meters and feet**. In **SRNAV**, the model must generate a route in the format **[blue, \<$action_1$\>, $road_1$, \<$action_2$\>, $road_2$, ..., \<$action_n$\>, $road_n$, red]**, under the assumption that travel begins in a vehicle facing toward the top of the map [2512.03558].

A notable methodological feature is the benchmark’s reliance on manual verification after automatic derivation. This suggests that pure extraction from vector data was not considered sufficient once rendered map visibility, label placement, and route readability entered the evaluation loop.

## 4. Evaluation protocol and benchmark results

CartoMapQA evaluates **15** vision-language models in **zero-shot** settings, including open-source and proprietary systems. The reported proprietary models include **GPT-4V**, **GPT-4o**, **o1**, **o3**, **Gemini 2.5 Pro**, and **Claude 3.7 Sonnet**. Open-source models include **InternVL 2.5**, **Qwen2.5-VL**, **LLaVA-OneVision**, **Llama vision models**, and **Llama 4 Scout**. Experiments were run on **four NVIDIA A100 80GB GPUs** [2512.03558].

The evaluation metrics are task-specific. **MFSEM** uses **Accuracy**. **STMF presence** uses **Accuracy** and **Macro F1**. **STMF/MTMF counting** uses **RMSE** and **$r^2$**, with **aRMSE** and **a$r^2$** reported for MTMF. **STMF name listing** uses **average precision**, **average recall**, and **average F1**; **MTMF** uses corresponding average micro metrics. **RLEST** uses **RMSE**, **MAPE**, and **$r^2$**. **MMLOC** uses **Accuracy**. **SRNAV** uses **Shortest Route Success Rate** \(denoted **$SR^2$**\), **Average Step Accuracy** \(**aSA**\), and **Connectivity** [2512.03558].

The reported results establish that performance remains far from robust. In **MFSEM**, the best model is **Gemini 2.5 Pro** with **0.652** overall accuracy, while the random baseline is **0.194**. The same section reports best category scores of **0.820** on symbols, **0.389** on areas, and **0.444** on lines, indicating that area and line semantics remain substantially harder than point-like symbols [2512.03558].

In **STMF**, **Gemini** leads the presence task with **Acc 0.843** and **MF1 0.842**, the counting task with **RMSE 1.367** and **$r^2 = 0.761$**, and the name-listing task with **aPrec 0.885**, **aRec 0.929**, and **aF1 0.894**. **MTMF** is clearly harder: its best counting result is **Gemini** with **aRMSE 0.582**, while best name-listing performance is **o1** with **amF1 0.712** [2512.03558].

The higher-level tasks show sharper brittleness. In **MMLOC**, the best model, **Gemini**, reaches only **0.600** accuracy overall. In **RLEST**, the strongest proprietary systems vary by condition, but **o1**, **o3**, **GPT-4o**, and **Gemini** dominate; the paper also notes that chain-of-thought prompting improved route-length estimation for most models. In **SRNAV**, the strongest model, **o3**, attains **$SR^2 = 0.338$**, **aSA 0.477**, and **Connectivity 0.378**, while **GPT-4o** reaches **$SR^2 = 0.029$** and **Connectivity 0.072**. Even the best exact shortest-route match therefore occurs only about **33.8%** of the time [2512.03558].

Taken together, these results support the benchmark’s principal argument: strong general multimodal competence does not imply strong cartographic competence.

## 5. Error modes, diagnostic value, and limitations

CartoMapQA’s diagnostic value lies partly in its explicit error analysis. The benchmark identifies **OCR failures** as a major weakness across tasks involving POI names, road names, and scale bars. In the analyzed subset of **STMF** name-listing errors, the main problems were **OCR errors** and **misrecognition of feature type**, while hallucinations were reported to be rare. This is a significant clarification because map-QA failure is not characterized primarily as open-ended fabrication; it is characterized more often as breakdown in reading, grounding, and semantic discrimination [2512.03558].

The benchmark also isolates **map-specific semantic confusion**. Models often confuse categories such as **restaurant**, **fast-food store**, and **coffee shop**. In **STMF**, counting performance deteriorates as the true count rises, and models tend to undercount once the map becomes denser. In **RLEST**, even when a strong model reads scale bars correctly in many cases, the paper reports examples of irrelevant reinterpretation into **inches or centimeters**, which the authors treat as evidence of incomplete grounding between the visual scale and the map task [2512.03558].

For **SRNAV**, the reported error types are explicitly topological. In an analysis of **30%** of incorrect routes from **o3**, the main categories were **incorrect origin/destination road name (22%)**, **incorrect direction changes (18%)**, **unconnected roads (26%)**, **one-way violations (2%)**, **redundant final steps after destination (15%)**, and **minor road name issues (17%)**. The dominant problem, **unconnected roads**, indicates that route generation fails not merely because of OCR, but because the model’s internal representation of road-network continuity remains weak [2512.03558].

The benchmark’s limitations are also sharply defined. It is restricted to **three cities** in **English-speaking countries**, relies on **OSM-based rendering**, and evaluates models in a **zero-shot** setting rather than through a train/validation/test protocol described in the paper. The authors also note that the task set is broad but not exhaustive and explicitly suggest future expansion toward more complex navigation and social-indicator estimation. A plausible implication is that CartoMapQA should be treated as a foundational benchmark rather than a complete map-understanding ontology.

## 6. Position within map-QA research

CartoMapQA sits within a rapidly expanding but heterogeneous literature on map question answering. Relative to the earlier **choropleth-focused “MapQA” dataset**, which contains **795,525 question-answer pairs** over **62,367 map images** and frames map QA as recovery of an underlying table from choropleth renderings, CartoMapQA broadens the scope from choropleths to a hierarchically structured benchmark centered on cartographic semantics, scale-bar interpretation, and route reasoning [2211.08545]. Relative to **MapIQ**, which introduces **14,706 question-answer pairs** across choropleth maps, cartograms, and proportional symbol maps and focuses on thematic map reading, CartoMapQA differs by emphasizing a progression from low-level feature semantics to map marker localization and shortest-route navigation rather than thematic-map analysis alone [2507.11625].

A different contrast appears when CartoMapQA is compared with structured geospatial QA benchmarks such as **“MapQA: Open-domain Geospatial Question Answering on Map Data”**, which provides **3,154 QA pairs** over **OpenStreetMap**-derived geometries and studies retrieval and Text-to-SQL approaches over geospatial databases. That benchmark is grounded in geometry-bearing entities and PostGIS reasoning rather than rendered cartographic images, so it addresses symbolic spatial computation more directly than visual map reading [2503.07871]. Historical-map GeoQA systems based on spatio-temporal knowledge graphs and LLM-mediated SPARQL generation extend this symbolic tradition into temporal cartography, showing how factual and descriptive QA can be grounded in structured features extracted from historical maps [2508.21491].

From a dataset-construction perspective, **MapQaTor** is relevant because it provides an annotation framework for API-grounded map QA with cached responses, explicit provenance, and JSON export, but it is an annotation system rather than a visual benchmark [2412.21015]. This suggests that CartoMapQA belongs to one branch of the field—benchmarking vision-language understanding of rendered cartographic maps—while structured geospatial QA and API-grounded QA occupy adjacent branches concerned with symbolic execution, tool use, and provenance.

The broader significance of CartoMapQA is therefore diagnostic. It makes visible a separation that had often been blurred: competence in reading maps as cartographic artifacts is not equivalent to competence in querying geospatial databases, and neither is equivalent to generic visual question answering. By structuring evaluation from isolated feature semantics to shortest-route navigation, the benchmark provides a concrete basis for analyzing where contemporary models fail, and for distinguishing OCR limitations, semantic confusions, and topological reasoning failures as separate research problems [2512.03558].

Source: https://www.emergentmind.com/topics/cartomapqa