VecMap-Bench: Unified Remote Sensing Mapping
- VecMap-Bench is a unified benchmark for remote sensing vector mapping that aggregates 54K high-resolution images and 800K vector instances across buildings, roads, and water bodies.
- It uses a GeoJSON-style Structured Vector Language (SVL) to encode geometry, semantics, and topology, enabling the generation of fully executable, GIS-compatible maps.
- The benchmark supports diverse evaluation protocols with metrics like mAP, IoU, and topology-aware scores, addressing both closed and network-style object representations.
Searching arXiv for VecMap-Bench and closely related work to ground the article in current literature. Search query: "VecMap-Bench VecLang remote sensing vector mapping" Search query: "site:arxiv.org VecMap-Bench" VecMap-Bench is a benchmark for unified multiclass vector mapping in remote sensing, introduced alongside the VecLang paradigm in "Vector Map as Language: Toward Unified Remote Sensing Vector Mapping" (Yan et al., 9 Jun 2026). It aggregates 54 000 high-resolution aerial and satellite image chips and roughly 800 000 vector instances across three core semantic categories—building footprints, road network segments (or polylines), and water-body polygons—drawn from nine public sources. Its defining feature is that every annotation is converted into a GeoJSON-style Structured Vector Language (SVL), so that models can be trained to generate fully executable, GIS-compatible maps rather than category-specific polygonal masks or graph outputs alone (Yan et al., 9 Jun 2026).
1. Problem setting and benchmark rationale
Remote sensing vector mapping aims to generate structured maps of geospatial entities, such as buildings, roads, and water bodies, from remote sensing imagery. In the formulation associated with VecMap-Bench, existing methods typically represent vector objects as polygons or graphs, making them suitable only for specific categories: polygons poorly capture topological relations, while graphs often blur instance boundaries (Yan et al., 9 Jun 2026). VecMap-Bench is designed as a single, large-scale, richly annotated benchmark for this setting.
The benchmark is explicitly oriented toward unified multiclass vector mapping rather than isolated single-category tasks. It therefore combines closed objects and network-style objects in one evaluation environment: building footprints and water-body polygons on the one hand, and road network segments on the other. A plausible implication is that the benchmark is intended to stress not only geometric fidelity but also representational interoperability across heterogeneous entity structures.
2. Dataset composition and split design
VecMap-Bench aggregates data across single-class, multiclass, cross-dataset, and open-vocabulary settings. Rather than impose a single split across all 54 k images, it preserves the original train/val/test partitions when available and augments them to yield balanced, category-stratified splits (Yan et al., 9 Jun 2026).
| Source | Setting | Split |
|---|---|---|
| WHU building dataset (~21 k chips) | buildings | 2 600 train, 500 val, 600 test |
| CityScale/Sat2Graph road dataset (~10 k chips) | roads | 7 500 train, 1 000 val, 1 500 test |
| Vec-WB waterbody dataset (~12 k chips) | water | 8 400/1 800/1 800 via 70/15/15 |
| IRSAMap multiclass dataset (~7 k chips) | multiclass | 5 600 train, 700 val, 700 test |
| CrowdAI buildings, SpaceNet roads | cross-dataset hold-out | 2 000/2 500 test chips |
| COCO, iSAID | open-vocabulary zero-shot | 500/500 chips |
Within each split, class distributions are roughly constant, with buildings at about 60% of instances in single-class, roads at about 30%, and water at about 10%. In total, the training pool comprises approximately 43 k images and approximately 640 k instances; validation comprises approximately 6 k images and approximately 80 k instances; and test comprises approximately 5 k images and approximately 80 k instances, plus the cross-dataset and open-vocabulary hold-outs (Yan et al., 9 Jun 2026).
The benchmark therefore covers not only conventional in-distribution evaluation but also transfer and zero-shot regimes. This suggests that VecMap-Bench is structured to probe both task competence and representational generalization.
3. Structured Vector Language and annotation schema
The core annotation format in VecMap-Bench is the Structured Vector Language. Each vector “Feature” in SVL follows a small JSON schema with four key fields: id, class, geometry, and optional topology (Yan et al., 9 Jun 2026). The id is a unique instance identifier; class is one of "building", "road", or "waterbody"; geometry is a GeoJSON geometry with a type such as "Polygon", "LineString", or "MultiLineString" and corresponding ordered vertex lists; and topology stores category-specific relations.
For buildings and water, topology may include holes, represented as arrays of inner-ring coordinate arrays. For roads, topology may include junctions, represented as arrays of objects of the form {pt: [x,y], connected_ids: [...]}. The schema is described as reversible and GeoJSON-like, and the benchmark provides .geojson SVL files together with an optional “map-to-language” converter that can reserialize them into plain SVL text prompts (Yan et al., 9 Jun 2026).
A minimal building entry is given in the benchmark description as a Feature whose properties contain an id such as "bldg_0203" and class "building", whose geometry is a Polygon, and whose topology contains "holes": []. A road vectorization unit is similarly represented as a Feature with class "road", geometry of type "MultiLineString", and topology-level junctions linking connected road identifiers. The significance of this design is that geometry, semantics, and topology are serialized in a common textual format suitable for vision-language generation.
4. Evaluation protocols and metrics
VecMap-Bench supports both polygonal and network-style evaluation (Yan et al., 9 Jun 2026). For closed objects, namely buildings and water, it reports instance-level mAP averaged over IoU thresholds , pixel-level IoU, contour-IoU (C-IoU), and PoLiS.
The benchmark defines instance-level mAP as
Pixel-level IoU is given by
C-IoU measures boundary overlap, while PoLiS, taken from Avbelj et al., penalizes excessive vertices. For network-style objects, namely roads, VecMap-Bench follows He et al.’s protocols and reports topology-aware Precision, Recall, F1, and APLS (average path length similarity) (Yan et al., 9 Jun 2026).
The released evaluation scripts cover mAP, IoU, C-IoU, PoLiS, Precision, Recall, F1, and APLS, and the README specifies that the code uses shapely and custom graph routines. This metric design is notable because it prevents a single geometric criterion from dominating assessment across categories with fundamentally different structural properties.
5. Baseline performance and generalization behavior
Across single-class tasks, VecLang is reported as yielding the strongest unified performance on VecMap-Bench (Yan et al., 9 Jun 2026). On WHU buildings, it attains 88.96 mAP, 92.22 IoU, 92.01 C-IoU, and 0.85 PoLiS, with a reported mAP over the best polygon method. On CityScale roads, it reaches 75.96% Recall, 73.95% F1, and 64.05 APLS, described as competitive with graph-based methods. On Vec-WB water, it achieves 64.82 mAP, 86.77 IoU, 76.88 C-IoU, and PoLiS 4.96.
In the multiclass IRSAMap setting, VecLang jointly outputs all three categories with 71.04 mAP for buildings, 85.12% precision for roads, and 50.65 mAP for water, outperforming Mask2Former, Mask R-CNN, UniVector, and ACPV-Net (Yan et al., 9 Jun 2026). Under cross-dataset transfer, a model trained on WHU and tested on CrowdAI obtains 17.81 mAP and 53.05 IoU, compared with 1–6 mAP for specialized methods, while CityScale to SpaceNet transfer yields 37.60% F1 and 32.97 APLS. In open-vocabulary zero-shot tests on unseen remote categories—plane, soccer field, pool, and tennis court—VecLang scores 78.22% mean, versus approximately 40% for top segmentation baselines.
These results are used to support the claim that structured text can serve as a universal language for maps. A plausible implication is that the benchmark is not merely evaluating vector extraction accuracy but also testing whether a shared textual representation can sustain cross-category modeling and out-of-domain transfer.
6. Position within the benchmark landscape, practical use, and terminology
VecMap-Bench is explicitly framed as advancing beyond prior benchmarks such as SpaceNet, DeepGlobe, WHU, and GID, which focus on a single category, use either polygon or graph outputs, and number only a few thousand images (Yan et al., 9 Jun 2026). By contrast, VecMap-Bench unifies scale, heterogeneity, and rich annotations: 54 000 images, approximately 800 000 instances, closed and network objects, multiclass and transfer settings, and topological relations such as holes and junctions in a reversible GeoJSON-like SVL schema.
The benchmark distribution includes all images, SVL annotations, train/val/test splits, and evaluation scripts. The prescribed usage pipeline is operationally simple: download the image chips and .geojson SVL files; optionally run the provided map-to-language converter; invoke the evaluation code via the README; split images using the CSV manifests; and fine-tune a vision-LLM, for example Qwen3-VL, under the two-stage recipe of LoRA SFT plus GRPO RL (Yan et al., 9 Jun 2026). Parsed vectors can then be inspected with GIS tools such as QGIS or used in downstream tasks including change detection, map updating, and network analysis.
A terminological ambiguity should be noted. In a distinct line of work on map entity representation learning, the VecCity library states that it “thus functions as ‘VecMap-Bench’ for map-entity representation learning,” standardizing data, methods, and evaluation in one unified framework (Zhang et al., 2024). This suggests that “VecMap-Bench” can also be used as a broader benchmark metaphor for map-centric ML infrastructure. In the stricter sense relevant here, however, VecMap-Bench denotes the remote sensing benchmark introduced with VecLang, centered on SVL-based vector map generation across buildings, roads, and water bodies (Yan et al., 9 Jun 2026).