CartoMapQA: Benchmark for Cartographic Map QA

• CartoMapQA is a benchmark for testing vision-language models on cartographic map interpretation, featuring hierarchically structured tasks.
• It comprises 2,251 questions over 853 maps, assessing skills from isolated feature semantics to complex route generation.
• The benchmark exposes model weaknesses in OCR, geospatial reasoning, and symbol extraction, driving future improvements in map understanding.

Searching arXiv for the CartoMapQA paper and closely related map-QA benchmarks to ground the article in current literature. CartoMapQA is a benchmark dataset and evaluation suite for testing how well large vision-LLMs understand cartographic maps through question-answering tasks. It is organized as a hierarchically structured map-understanding benchmark spanning low-, mid-, and high-level skills, and each sample contains either a cartographic map image or an isolated map-feature image together with a question or request and a ground-truth answer. The dataset contains 2,251 questions over 853 OpenStreetMap-derived maps, covers six tasks grouped under map information understanding, scale and distance interpretation, and directional reasoning / navigation, and was introduced to expose persistent weaknesses of contemporary vision-LLMs in map-specific semantics, OCR on map text, and geospatial reasoning (Ung et al., 3 Dec 2025).

1. Definition and scope

CartoMapQA is defined as a question-answering benchmark for cartographic maps rather than a generic visual question answering resource. Its samples combine a map image or isolated map-feature image with a question and a ground-truth answer, and its task design explicitly targets map-specific operations such as symbol recognition, embedded information extraction, scale interpretation, marker localization, and route-based reasoning (Ung et al., 3 Dec 2025).

The benchmark is motivated by the claim that cartographic maps occupy a distinctive position among geospatial representations. In the benchmark description, text-only inputs are described as limited by tokenization and often unable to preserve rich spatial structure, while satellite imagery is described as visually rich but cluttered with irrelevant detail. Cartographic maps, by contrast, provide a structured and semantically abstracted representation through standardized symbols, color-coded areas, roads and route structure, textual labels, and scale bars. This framing is important because it makes CartoMapQA a benchmark for cartographic interpretation as such, not merely for generic OCR or image captioning (Ung et al., 3 Dec 2025).

The paper presents CartoMapQA as a diagnostic benchmark rather than a saturated leaderboard. Its central claim is that current multimodal models can appear strong on general multimodal reasoning while still failing at map-specific operations. A plausible implication is that map understanding should be evaluated as a hierarchy of dependent competencies rather than as a single end-to-end score.

2. Dataset composition and task hierarchy

CartoMapQA contains 2,251 questions distributed across six tasks: MFSEM, STMF, MTMF, RLEST, MMLOC, and SRNAV. These tasks are organized to separate foundational perceptual competence from higher-order geospatial reasoning, so that failures in navigation can be traced back to weaker capabilities such as symbol interpretation, label extraction, or marker grounding (Ung et al., 3 Dec 2025).

Task	Questions	Core operation
MFSEM	463	Isolated map-feature semantics
STMF	510	Single-type feature presence, counting, name listing
MTMF	150	Multiple-type feature counting and name listing
RLEST	600	Route length estimation with scale bars
MMLOC	250	Marker localization by road-name intersection
SRNAV	278	Shortest-route instruction generation

The hierarchy is explicit. MFSEM tests isolated feature semantics without broader map context. STMF and MTMF move to full-map feature analysis. MMLOC serves as a bridge from map reading to route reasoning by requiring a marker to be grounded to two intersecting road names. RLEST tests the use of scale bars for quantitative estimation. SRNAV is the highest-level task and asks for a shortest drivable route in a structured turn-by-turn format (Ung et al., 3 Dec 2025).

The answer formats are deliberately mixed. MFSEM is multiple-choice, while the remaining tasks are predominantly open-ended. Across the whole benchmark, 463 questions (21%) are multiple-choice and 1,788 (79%) are open-ended. This matters because the benchmark does not collapse map QA into a single response regime; instead it spans classification, binary decisions, counting, name listing, numeric estimation, and structured route generation (Ung et al., 3 Dec 2025).

The benchmark also specifies image and map distributions. Rendered maps have size 1366 × 768 pixels. The maps use zoom levels 15, 16, 17, 18, 19, with counts 1, 7, 213, 187, 445 respectively. The data come from San Jose, Manchester, and Melbourne, chosen from three English-speaking countries in order to reduce confounding from multilingual text comprehension (Ung et al., 3 Dec 2025).

3. Construction methodology and ground-truth generation

CartoMapQA is built from OpenStreetMap rather than proprietary services, and the paper explicitly cites licensing openness and reproducibility as reasons for this design choice. Map rendering uses Folium, while graph-based path computation and related operations use OSMnx and NetworkX. This makes the benchmark technically reproducible and aligns it with open data workflows (Ung et al., 3 Dec 2025).

The dataset construction pipeline differs by task. MFSEM uses isolated OpenStreetMap feature images. Its distractor choices are created through a combination of CLIP image embeddings and semantic class structure: two most similar features from the same class, one most similar feature from a different class, and one randomly selected feature, together with the correct label, yielding five options whose order is randomly shuffled. This design makes the multiple-choice setting nontrivial by avoiding obviously unrelated distractors (Ung et al., 3 Dec 2025).

For STMF and MTMF, the authors render full cartographic maps from OSM, extract map-feature data within each map boundary, and then manually verify counts and names because rendered content may diverge from underlying OSM extraction due to rendering limitations, outdated entries, or label mismatches. For RLEST, MMLOC, and SRNAV, ground truth is derived with OSMnx + NetworkX and then manually checked where needed. The paper states that routes in SRNAV were manually verified and corrected for consistency with rendered maps, and that 300 initial routes were reduced to 278 valid routes because some were filtered out for readability issues such as overlapping map elements (Ung et al., 3 Dec 2025).

The benchmark’s tasks encode several exact conventions. In MMLOC, the answer is the pair of road names intersecting at a colored marker. In RLEST, maps include two blue markers, a predefined blue path, and a scale bar shown in meters and feet. In SRNAV, the model must generate a route in the format [blue, <$action_1$>, $road_1$, <$action_2$>, $road_2$, ..., <$action_n$>, $road_n$, red], under the assumption that travel begins in a vehicle facing toward the top of the map (Ung et al., 3 Dec 2025).

A notable methodological feature is the benchmark’s reliance on manual verification after automatic derivation. This suggests that pure extraction from vector data was not considered sufficient once rendered map visibility, label placement, and route readability entered the evaluation loop.

4. Evaluation protocol and benchmark results

CartoMapQA evaluates 15 vision-LLMs in zero-shot settings, including open-source and proprietary systems. The reported proprietary models include GPT-4V, GPT-4o, o1, o3, Gemini 2.5 Pro, and Claude 3.7 Sonnet. Open-source models include InternVL 2.5, Qwen2.5-VL, LLaVA-OneVision, Llama vision models, and Llama 4 Scout. Experiments were run on four NVIDIA A100 80GB GPUs (Ung et al., 3 Dec 2025).

The evaluation metrics are task-specific. MFSEM uses Accuracy. STMF presence uses Accuracy and Macro F1. STMF/MTMF counting uses RMSE and $r^2$, with aRMSE and a$r^2$ reported for MTMF. STMF name listing uses average precision, average recall, and average F1; MTMF uses corresponding average micro metrics. RLEST uses RMSE, MAPE, and $r^2$. MMLOC uses Accuracy. SRNAV uses Shortest Route Success Rate $denoted **$, Average Step Accuracy $road_1$0, and Connectivity (Ung et al., 3 Dec 2025).

The reported results establish that performance remains far from robust. In MFSEM, the best model is Gemini 2.5 Pro with 0.652 overall accuracy, while the random baseline is 0.194. The same section reports best category scores of 0.820 on symbols, 0.389 on areas, and 0.444 on lines, indicating that area and line semantics remain substantially harder than point-like symbols (Ung et al., 3 Dec 2025).

In STMF, Gemini leads the presence task with Acc 0.843 and MF1 0.842, the counting task with RMSE 1.367 and $road_1$1, and the name-listing task with aPrec 0.885, aRec 0.929, and aF1 0.894. MTMF is clearly harder: its best counting result is Gemini with aRMSE 0.582, while best name-listing performance is o1 with amF1 0.712 (Ung et al., 3 Dec 2025).

The higher-level tasks show sharper brittleness. In MMLOC, the best model, Gemini, reaches only 0.600 accuracy overall. In RLEST, the strongest proprietary systems vary by condition, but o1, o3, GPT-4o, and Gemini dominate; the paper also notes that chain-of-thought prompting improved route-length estimation for most models. In SRNAV, the strongest model, o3, attains $road_1$2, aSA 0.477, and Connectivity 0.378, while GPT-4o reaches $road_1$3 and Connectivity 0.072. Even the best exact shortest-route match therefore occurs only about 33.8% of the time (Ung et al., 3 Dec 2025).

Taken together, these results support the benchmark’s principal argument: strong general multimodal competence does not imply strong cartographic competence.

5. Error modes, diagnostic value, and limitations

CartoMapQA’s diagnostic value lies partly in its explicit error analysis. The benchmark identifies OCR failures as a major weakness across tasks involving POI names, road names, and scale bars. In the analyzed subset of STMF name-listing errors, the main problems were OCR errors and misrecognition of feature type, while hallucinations were reported to be rare. This is a significant clarification because map-QA failure is not characterized primarily as open-ended fabrication; it is characterized more often as breakdown in reading, grounding, and semantic discrimination (Ung et al., 3 Dec 2025).

The benchmark also isolates map-specific semantic confusion. Models often confuse categories such as restaurant, fast-food store, and coffee shop. In STMF, counting performance deteriorates as the true count rises, and models tend to undercount once the map becomes denser. In RLEST, even when a strong model reads scale bars correctly in many cases, the paper reports examples of irrelevant reinterpretation into inches or centimeters, which the authors treat as evidence of incomplete grounding between the visual scale and the map task (Ung et al., 3 Dec 2025).

For SRNAV, the reported error types are explicitly topological. In an analysis of 30% of incorrect routes from o3, the main categories were incorrect origin/destination road name (22%), incorrect direction changes (18%), unconnected roads (26%), one-way violations (2%), redundant final steps after destination (15%), and minor road name issues (17%). The dominant problem, unconnected roads, indicates that route generation fails not merely because of OCR, but because the model’s internal representation of road-network continuity remains weak (Ung et al., 3 Dec 2025).

The benchmark’s limitations are also sharply defined. It is restricted to three cities in English-speaking countries, relies on OSM-based rendering, and evaluates models in a zero-shot setting rather than through a train/validation/test protocol described in the paper. The authors also note that the task set is broad but not exhaustive and explicitly suggest future expansion toward more complex navigation and social-indicator estimation. A plausible implication is that CartoMapQA should be treated as a foundational benchmark rather than a complete map-understanding ontology.

6. Position within map-QA research

CartoMapQA sits within a rapidly expanding but heterogeneous literature on map question answering. Relative to the earlier choropleth-focused “MapQA” dataset, which contains 795,525 question-answer pairs over 62,367 map images and frames map QA as recovery of an underlying table from choropleth renderings, CartoMapQA broadens the scope from choropleths to a hierarchically structured benchmark centered on cartographic semantics, scale-bar interpretation, and route reasoning (Chang et al., 2022). Relative to MapIQ, which introduces 14,706 question-answer pairs across choropleth maps, cartograms, and proportional symbol maps and focuses on thematic map reading, CartoMapQA differs by emphasizing a progression from low-level feature semantics to map marker localization and shortest-route navigation rather than thematic-map analysis alone (Srivastava et al., 15 Jul 2025).

A different contrast appears when CartoMapQA is compared with structured geospatial QA benchmarks such as “MapQA: Open-domain Geospatial Question Answering on Map Data”, which provides 3,154 QA pairs over OpenStreetMap-derived geometries and studies retrieval and Text-to-SQL approaches over geospatial databases. That benchmark is grounded in geometry-bearing entities and PostGIS reasoning rather than rendered cartographic images, so it addresses symbolic spatial computation more directly than visual map reading (Li et al., 10 Mar 2025). Historical-map GeoQA systems based on spatio-temporal knowledge graphs and LLM-mediated SPARQL generation extend this symbolic tradition into temporal cartography, showing how factual and descriptive QA can be grounded in structured features extracted from historical maps (Liu et al., 29 Aug 2025).

From a dataset-construction perspective, MapQaTor is relevant because it provides an annotation framework for API-grounded map QA with cached responses, explicit provenance, and JSON export, but it is an annotation system rather than a visual benchmark (Dihan et al., 2024). This suggests that CartoMapQA belongs to one branch of the field—benchmarking vision-language understanding of rendered cartographic maps—while structured geospatial QA and API-grounded QA occupy adjacent branches concerned with symbolic execution, tool use, and provenance.

The broader significance of CartoMapQA is therefore diagnostic. It makes visible a separation that had often been blurred: competence in reading maps as cartographic artifacts is not equivalent to competence in querying geospatial databases, and neither is equivalent to generic visual question answering. By structuring evaluation from isolated feature semantics to shortest-route navigation, the benchmark provides a concrete basis for analyzing where contemporary models fail, and for distinguishing OCR limitations, semantic confusions, and topological reasoning failures as separate research problems (Ung et al., 3 Dec 2025).