---
title: 'VecMap-Bench: Unified Remote Sensing Mapping'
url: https://www.emergentmind.com/topics/vecmap-bench
type: topic
---

# VecMap-Bench: Unified Remote Sensing Mapping

Searching arXiv for VecMap-Bench and closely related work to ground the article in current literature.
Search query: "VecMap-Bench VecLang remote sensing vector mapping"
Search query: "site:arxiv.org VecMap-Bench"
VecMap-Bench is a benchmark for unified multiclass vector mapping in remote sensing, introduced alongside the VecLang paradigm in "Vector Map as Language: Toward Unified Remote Sensing Vector Mapping" [2606.10701]. It aggregates 54 000 high-resolution aerial and satellite image chips and roughly 800 000 vector instances across three core semantic categories—building footprints, road network segments (or polylines), and water-body polygons—drawn from nine public sources. Its defining feature is that every annotation is converted into a GeoJSON-style Structured Vector Language (SVL), so that models can be trained to generate fully executable, GIS-compatible maps rather than category-specific polygonal masks or graph outputs alone [2606.10701].

## 1. Problem setting and benchmark rationale

Remote sensing vector mapping aims to generate structured maps of geospatial entities, such as buildings, roads, and water bodies, from remote sensing imagery. In the formulation associated with VecMap-Bench, existing methods typically represent vector objects as polygons or graphs, making them suitable only for specific categories: polygons poorly capture topological relations, while graphs often blur instance boundaries [2606.10701]. VecMap-Bench is designed as a single, large-scale, richly annotated benchmark for this setting.

The benchmark is explicitly oriented toward unified multiclass vector mapping rather than isolated single-category tasks. It therefore combines closed objects and network-style objects in one evaluation environment: building footprints and water-body polygons on the one hand, and road network segments on the other. A plausible implication is that the benchmark is intended to stress not only geometric fidelity but also representational interoperability across heterogeneous entity structures.

## 2. Dataset composition and split design

VecMap-Bench aggregates data across single-class, multiclass, cross-dataset, and open-vocabulary settings. Rather than impose a single split across all 54 k images, it preserves the original train/val/test partitions when available and augments them to yield balanced, category-stratified splits [2606.10701].

| Source | Setting | Split |
|---|---|---|
| WHU building dataset (~21 k chips) | buildings | 2 600 train, 500 val, 600 test |
| CityScale/Sat2Graph road dataset (~10 k chips) | roads | 7 500 train, 1 000 val, 1 500 test |
| Vec-WB waterbody dataset (~12 k chips) | water | 8 400/1 800/1 800 via 70/15/15 |
| IRSAMap multiclass dataset (~7 k chips) | multiclass | 5 600 train, 700 val, 700 test |
| CrowdAI buildings, SpaceNet roads | cross-dataset hold-out | 2 000/2 500 test chips |
| COCO, iSAID | open-vocabulary zero-shot | 500/500 chips |

Within each split, class distributions are roughly constant, with buildings at about 60% of instances in single-class, roads at about 30%, and water at about 10%. In total, the training pool comprises approximately 43 k images and approximately 640 k instances; validation comprises approximately 6 k images and approximately 80 k instances; and test comprises approximately 5 k images and approximately 80 k instances, plus the cross-dataset and open-vocabulary hold-outs [2606.10701].

The benchmark therefore covers not only conventional in-distribution evaluation but also transfer and zero-shot regimes. This suggests that VecMap-Bench is structured to probe both task competence and representational generalization.

## 3. Structured Vector Language and annotation schema

The core annotation format in VecMap-Bench is the Structured Vector Language. Each vector “Feature” in SVL follows a small JSON schema with four key fields: `id`, `class`, `geometry`, and optional `topology` [2606.10701]. The `id` is a unique instance identifier; `class` is one of `"building"`, `"road"`, or `"waterbody"`; `geometry` is a GeoJSON geometry with a `type` such as `"Polygon"`, `"LineString"`, or `"MultiLineString"` and corresponding ordered vertex lists; and `topology` stores category-specific relations.

For buildings and water, topology may include `holes`, represented as arrays of inner-ring coordinate arrays. For roads, topology may include `junctions`, represented as arrays of objects of the form `{pt: [x,y], connected_ids: [...]}`. The schema is described as reversible and GeoJSON-like, and the benchmark provides `.geojson` SVL files together with an optional “map-to-language” converter that can reserialize them into plain SVL text prompts [2606.10701].

A minimal building entry is given in the benchmark description as a `Feature` whose `properties` contain an `id` such as `"bldg_0203"` and `class` `"building"`, whose `geometry` is a `Polygon`, and whose `topology` contains `"holes": []`. A road vectorization unit is similarly represented as a `Feature` with `class` `"road"`, `geometry` of type `"MultiLineString"`, and topology-level `junctions` linking connected road identifiers. The significance of this design is that geometry, semantics, and topology are serialized in a common textual format suitable for vision-language generation.

## 4. Evaluation protocols and metrics

VecMap-Bench supports both polygonal and network-style evaluation [2606.10701]. For closed objects, namely buildings and water, it reports instance-level mAP averaged over IoU thresholds \(T=\{0.50:0.05:0.95\}\), pixel-level IoU, contour-IoU (C-IoU), and PoLiS.

The benchmark defines instance-level mAP as

$$
\mathrm{mAP}=\frac{1}{|T|}\sum_{\tau\in T}\frac{\mathrm{TP}(\tau)}{\mathrm{TP}(\tau)+\mathrm{FP}(\tau)}.
$$

Pixel-level IoU is given by

$$
\mathrm{IoU}=\frac{|\hat{M}\cap M^*|}{|\hat{M}\cup M^*|}.
$$

C-IoU measures boundary overlap, while PoLiS, taken from Avbelj et al., penalizes excessive vertices. For network-style objects, namely roads, VecMap-Bench follows He et al.’s protocols and reports topology-aware Precision, Recall, F1, and APLS (average path length similarity) [2606.10701].

The released evaluation scripts cover mAP, IoU, C-IoU, PoLiS, Precision, Recall, F1, and APLS, and the README specifies that the code uses `shapely` and custom graph routines. This metric design is notable because it prevents a single geometric criterion from dominating assessment across categories with fundamentally different structural properties.

## 5. Baseline performance and generalization behavior

Across single-class tasks, VecLang is reported as yielding the strongest unified performance on VecMap-Bench [2606.10701]. On WHU buildings, it attains 88.96 mAP, 92.22 IoU, 92.01 C-IoU, and 0.85 PoLiS, with a reported \( \triangle +9 \) mAP over the best polygon method. On CityScale roads, it reaches 75.96% Recall, 73.95% F1, and 64.05 APLS, described as competitive with graph-based methods. On Vec-WB water, it achieves 64.82 mAP, 86.77 IoU, 76.88 C-IoU, and PoLiS 4.96.

In the multiclass IRSAMap setting, VecLang jointly outputs all three categories with 71.04 mAP for buildings, 85.12% precision for roads, and 50.65 mAP for water, outperforming Mask2Former, Mask R-CNN, UniVector, and ACPV-Net [2606.10701]. Under cross-dataset transfer, a model trained on WHU and tested on CrowdAI obtains 17.81 mAP and 53.05 IoU, compared with 1–6 mAP for specialized methods, while CityScale to SpaceNet transfer yields 37.60% F1 and 32.97 APLS. In open-vocabulary zero-shot tests on unseen remote categories—plane, soccer field, pool, and tennis court—VecLang scores 78.22% mean, versus approximately 40% for top segmentation baselines.

These results are used to support the claim that structured text can serve as a universal language for maps. A plausible implication is that the benchmark is not merely evaluating vector extraction accuracy but also testing whether a shared textual representation can sustain cross-category modeling and out-of-domain transfer.

## 6. Position within the benchmark landscape, practical use, and terminology

VecMap-Bench is explicitly framed as advancing beyond prior benchmarks such as SpaceNet, DeepGlobe, WHU, and GID, which focus on a single category, use either polygon or graph outputs, and number only a few thousand images [2606.10701]. By contrast, VecMap-Bench unifies scale, heterogeneity, and rich annotations: 54 000 images, approximately 800 000 instances, closed and network objects, multiclass and transfer settings, and topological relations such as holes and junctions in a reversible GeoJSON-like SVL schema.

The benchmark distribution includes all images, SVL annotations, train/val/test splits, and evaluation scripts. The prescribed usage pipeline is operationally simple: download the image chips and `.geojson` SVL files; optionally run the provided map-to-language converter; invoke the evaluation code via the README; split images using the CSV manifests; and fine-tune a vision-language model, for example Qwen3-VL, under the two-stage recipe of LoRA SFT plus GRPO RL [2606.10701]. Parsed vectors can then be inspected with GIS tools such as QGIS or used in downstream tasks including change detection, map updating, and network analysis.

A terminological ambiguity should be noted. In a distinct line of work on map entity representation learning, the VecCity library states that it “thus functions as ‘VecMap-Bench’ for map-entity representation learning,” standardizing data, methods, and evaluation in one unified framework [2411.00874]. This suggests that “VecMap-Bench” can also be used as a broader benchmark metaphor for map-centric ML infrastructure. In the stricter sense relevant here, however, VecMap-Bench denotes the remote sensing benchmark introduced with VecLang, centered on SVL-based vector map generation across buildings, roads, and water bodies [2606.10701].

Source: https://www.emergentmind.com/topics/vecmap-bench