SVM-City: Outdoor Multimodal City Dataset

• SVM-City is an outdoor, city-level dataset that enables multimodal instruction tuning by integrating data from satellite, aerial, drone, and vehicular sensors for comprehensive scene understanding.
• It spans 13 global cities and uses semantic scene graphs and ChatGPT-generated QA pairs to address limitations of indoor-focused LVLMs and single-modality datasets.
• City-VLM trained on SVM-City demonstrates significant performance gains on outdoor QA benchmarks by leveraging a VAE-based incomplete multimodal fusion module for robust missing-modality handling.

Searching arXiv for the primary paper on SVM-City and closely related records. SVM-City is an outdoor, city-level dataset explicitly designed for multiScale, multiView, and multiModal instruction tuning. The acronym denotes multiScale, multiView, multiModal City, and the dataset was introduced together with the LVLM City-VLM to address outdoor scene understanding under multidomain perception, where existing large vision-LLMs had primarily concentrated on indoor household scenes, single visual modalities, and building-scale contexts from humanoid viewpoints. In the formulation reported in "City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning" (Sun et al., 17 Jul 2025), SVM-City is positioned as the first multidomain perception outdoor scene understanding dataset, spanning satellite, aerial, drone, and vehicular sensing, and supporting instruction tuning for multimodal question answering across terrestrial, low-altitude, and high-altitude views.

1. Definition and design objectives

SVM-City was created to fill two limitations identified for prior LVLMs in scene understanding. First, outdoor scenarios involve larger-scale environments observed through various sensors from multiple viewpoints, including bird view and terrestrial view, whereas existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. The dataset therefore targets space–sky–land multidomain perception at bird’s-eye, low-altitude, and terrestrial views, with explicit support for missing-modality conditions through the downstream incomplete multimodal learning framework used by City-VLM (Sun et al., 17 Jul 2025).

The design is instruction-tuning oriented rather than a conventional static benchmark split. The paper constructs SVM-City for instruction tuning of City-VLM and does not report an official train/val/test split. Instead, City-VLM is trained on SVM-City and then evaluated on external outdoor QA benchmarks, namely EarthVQA, City-3DQA, and NuScenes-QA. This makes SVM-City closer to a foundation dataset for outdoor multimodal instruction tuning than to a narrowly scoped evaluation corpus.

A central implication of this design is that SVM-City is organized around scene understanding tasks expressed as free-form natural-language question answering. The data model is therefore not limited to object labels or pixelwise annotations; it encodes spatial relations, measurement-like queries, functional reasoning, and logical comparison questions intended to exercise outdoor perception across multiple sensing regimes.

2. Dataset composition, geographic coverage, and annotation pipeline

The dataset covers 13 cities across three regions: North America, Western Europe, and East Asia. The listed cities are New York, Boston, Portland, Philadelphia, Birmingham, Cambridge, Qingdao, Nanjing, Wuhan, Wuxi, Wuhu, Shenzhen, and Singapore. The sources include LoveDA, EarthExplorer, UrbanBIS, SensatUrban, and NuScenes, thereby combining remote sensing, photogrammetric, and autonomous-driving data into a single instruction-tuning resource (Sun et al., 17 Jul 2025).

Aspect	Reported content
Cities	13 cities across three regions
Images	420k
3D points	“4,811M” or “4810.8 million”
QA pairs	567k
Platforms	vehicle, drone, aerial plane, satellite

The paper reports two notations for the 3D scale: “4,811M” points and “4810.8 million” points. It explicitly states that both notations refer to the total number of points rather than the number of files, and that the difference is a rounding or formatting difference. The imagery count is 420k, and the dataset contains 567k QA pairs.

The annotation schema uses four outdoor spatial question types inspired by cognitive science: Localization, Measurement, Functionality, and Logicality. Localization addresses existence and spatial arrangement; Measurement targets size, shape, and quantity; Functionality targets purpose or affordance; Logicality targets relative relations and logical reasoning. The reported question distribution is Localization 26.55%, Measurement 16.54%, Functionality 24.02%, and Logicality 32.89%. The modality distribution is point-only 41.36%, image-only 16.38%, and point+image 42.26%.

The annotation pipeline proceeds by segmenting objects with existing 2D and 3D segmentation methods such as HRNet and B-Seg, constructing scene graphs that capture objects and relations, adding manual attributes including color, pose, and function, and then using ChatGPT to generate diverse QA pairs from triples and templates. The answer format is free-form natural language, and the dataset language is polished by ChatGPT for grammatical correctness. This suggests that SVM-City is intentionally structured around semantic graph abstraction rather than around direct sensor-to-language alignment alone.

3. Modalities, viewpoints, and cross-source alignment

SVM-City integrates both 2D RGB imagery and 3D point clouds. The 2D sources are high-altitude satellite remote sensing images from LoveDA, aerial orthophotos from EarthExplorer, and vehicle-mounted cameras from NuScenes. The 3D sources are low-altitude drone point clouds from UrbanBIS and SensatUrban, together with vehicular LiDAR from NuScenes. The corresponding viewpoints are terrestrial view, low-altitude view, and high-altitude view, with scales described respectively as single drive less than 1 km, community-scale approximately 5–20 km, and metropolitan-scale hundreds to thousands of km (Sun et al., 17 Jul 2025).

A critical design choice is that cross-view and multi-source alignment is not defined by universal pixel-level or point-level registration across the whole compiled corpus. Within a single dataset such as NuScenes, standard camera–LiDAR calibration is available. Across SVM-City’s multi-source city-level compilation, however, cross-view and multi-source data are fused at the semantic level via scene graphs and QA templates rather than through pixel/point-level correspondences across views. This is an important technical distinction because it limits what kinds of correspondence reasoning are directly supervised.

The paper also notes that the model processes inputs at fixed resolution in the vision encoders to ensure stable positional encoding, while exact sensor resolution details are dataset-specific and not enumerated. Accordingly, the alignment mechanism in the associated model depends on dedicated encoders and probabilistic fusion, not on a single canonical georegistered coordinate system spanning all sources. A plausible implication is that SVM-City emphasizes semantic consistency across platforms more than strict geometric equivalence.

4. City-VLM architecture and incomplete multimodal learning

City-VLM is the LVLM designed for SVM-City, and its architecture consists of EVA-CLIP-E for 2D images, Uni3D-L for 3D point clouds, Vicuna-7B as the LLM with LoRA fine-tuning, and an Incomplete Multimodal Fusion Module (IMF). The IMF is described as a VAE-based probabilistic fusion mechanism that constructs a joint distribution over modalities rather than relying on explicit fusion operations such as concatenation. This design is intended to make the model robust when certain modalities are missing or corrupted (Sun et al., 17 Jul 2025).

The core input and generation formulation is stated as

$\mathbf{z} = \text{IMF}(\mathbf{x_v}),$

and the autoregressive answer model is

$$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$

Feature extraction for the two visual streams is written as

$$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$

The IMF constructs a Gaussian probabilistic embedding:

$$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$

with parameterization

$$\boldsymbol{\mu} = f_{\mu}(\mathbf{r_z}), \qquad \log(\boldsymbol{\sigma})=f_{\sigma}(\mathbf{r_z}),$$

and reparameterization

$$\mathbf{z} = \boldsymbol{\mu}+\boldsymbol{\varepsilon} \cdot \boldsymbol{\sigma}, \qquad \boldsymbol{\varepsilon} \sim \mathcal{N}(0,1).$$

The regularizer is

$$L_{kl} = KL[\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2) \| \mathcal{N}(0, \mathbf{I})] =-\frac{1}{2} (1+\log (\boldsymbol{\sigma}^2)-\boldsymbol{\mu}^2-\boldsymbol{\sigma}^2 ).$$

Missing modality handling is implemented by padding the absent 2D or 3D input with zeros to maintain shape consistency, after which the IMF models the missing modality through learnable distribution parameters. The training objective combines the autoregressive language modeling objective with the KL regularization above. No additional variational bounds or explicit likelihoods beyond what is shown are presented. In implementation terms, the reported setup uses Adam, weight decay $5\mathrm{e}{-4}$, learning rate $1\mathrm{e}{-3}$, batch size 4 per GPU, PyTorch 2.0.1, CUDA 11.8, and $8\times$ NVIDIA RTX A6000.

5. Benchmark tasks and reported empirical performance

City-VLM trained on SVM-City is evaluated on three representative outdoor QA benchmarks: EarthVQA for high-altitude VQA, City-3DQA for low-altitude 3D QA, and NuScenes-QA for terrestrial multimodal VQA. EarthVQA contains 1,809 test images with 63,225 QA pairs; City-3DQA uses sentence-wise and city-wise test sets with 78k QA pairs each; NuScenes-QA uses 83k QA pairs with 390k LiDAR point clouds and 1.4M camera images. For EarthVQA and NuScenes-QA, accuracy is reported per category and overall. For City-3DQA, accuracy is reported for single-hop and multi-hop, sentence-wise and city-wise splits. For auto-regressive outputs, the paper uses a GPT-4 judge to compare generated answers with ground-truth semantics, following the LLaVA evaluation protocol (Sun et al., 17 Jul 2025).

The principal aggregate claim is that City-VLM achieves 18.14% average performance improvement over existing LVLMs on outdoor QA tasks. On EarthVQA, City-VLM with IMF reports overall accuracy $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$0, compared with SOBA at $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$1, Instruct-BLIP at $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$2, and BLIP-2 at $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$3. The paper also reports category-leading scores for City-VLM with IMF in Bas Ju $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$4, Rel Ju $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$5, Bas Co $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$6, Rel Co $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$7, Obj An $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$8, and Com An $$\text{P}(\mathbf{a}|\mathbf{h_v},\mathbf{h_q}) = \prod_{j=1}^{L}\text{P}(a_j|\mathbf{h_v},\mathbf{h_q},a_{<j}).$$9. In ablation, IMF outperforms MLP fusion at $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$0 OA and Attention at $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$1 OA.

On City-3DQA, City-VLM with IMF reaches $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$2 on the sentence-wise setting and $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$3 on the city-wise setting, exceeding Sg-CityU by $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$4 and $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$5, respectively. The paper further states that general LVLMs such as Qwen-VL and LLaVA, which rely on 2D projections of point clouds, remain at approximately $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$6 accuracy, indicating difficulty with low-altitude 3D scene understanding.

On NuScenes-QA, City-VLM with IMF reports $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$7 for camera-only, $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$8 for LiDAR-only, and $$\mathbf{r_i} = Encoder_i(\mathbf{x_i}), \qquad \mathbf{r_p} = Encoder_p(\mathbf{x_p}).$$9 for camera+LiDAR. The corresponding baselines are BEVDet+MCAN at $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$0, CenterPoint+MCAN at $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$1, and MSMDFusion+MCAN at $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$2. The reported per-type gains are Exist $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$3 $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$4, Object $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$5 $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$6, Status $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$7 $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$8, and Comparison $$p(\mathbf{r_z} \mid \mathbf{r_i}, \mathbf{r_p})=\mathcal{N}(\mathbf{r_z};\boldsymbol{\mu},\boldsymbol{\sigma}^2),$$9 $$\boldsymbol{\mu} = f_{\mu}(\mathbf{r_z}), \qquad \log(\boldsymbol{\sigma})=f_{\sigma}(\mathbf{r_z}),$$0. Counting is reported as relatively weaker, which the paper associates with autoregressive counting limitations.

6. Limitations, applications, and research significance

The paper identifies several limitations. Significant differences between aerial, low-altitude, and terrestrial views make cross-view reasoning challenging; SVM-City fuses semantics largely via scene graphs and instruction tuning rather than strict geometric alignment, which may limit fine-grained correspondence reasoning. Counting and measurement remain difficult because autoregressive generation struggles on counting-related questions. The QA construction pipeline may introduce label noise and bias because it depends on automated segmentation-driven object extraction and ChatGPT-based question generation. Missing modality handling by zero-padding is characterized as pragmatic rather than definitive, with more sophisticated imputation or cross-modal generation suggested as future directions (Sun et al., 17 Jul 2025).

The reported applications are correspondingly broad. The dataset and model are proposed for autonomous driving, urban planning and digital cities, remote sensing, robotics, and multimodal geospatial analytics. In these settings, SVM-City supplies a unified supervision source for reasoning over buildings, infrastructure, land use, object relations, and navigation-relevant semantics across sensor modalities and spatial scales.

Its broader significance lies in establishing a multimodal instruction-tuning substrate for outdoor scene understanding that is city-scale, cross-view, and incomplete-modality aware. Because the dataset is organized around scene graphs, natural-language QA, and multidomain sensing, it offers a different operating point from datasets centered solely on dense geometry or single-view image-language alignment. This suggests a shift toward outdoor LVLM training regimes in which semantic abstraction, multimodal robustness, and view diversity are treated as first-class requirements rather than auxiliary augmentations.