3DCity-LLM-1.2M: Urban 3D Vision-Language Data

• 3DCity-LLM-1.2M is a large-scale urban 3D vision-language dataset that integrates text, 2D imagery, and 3D point clouds for multimodal perception.
• It combines explicit numerical grounding with long-form, instruction-based QA pairs to enable fine-grained object analysis, spatial reasoning, and scene planning.
• The dataset employs a VLM-driven generation pipeline and rigorous quality control to ensure reliable, context-sensitive supervision rooted in real city data.

Searching arXiv for the specified dataset paper and a closely related urban 3D dataset paper to ground the article in current preprints. 3DCity-LLM-1.2M is a city-scale 3D vision-language dataset introduced with the unified 3DCity-LLM framework for multimodal perception and understanding in urban environments (Chen et al., 24 Mar 2026). It was created to support training and evaluation of multimodal LLMs on city-scale tasks that require 3D spatial structure, explicit numerical grounding, and user-oriented contextual simulation. The dataset comprises approximately 1.2 million high-quality samples across seven representative task categories, spanning fine-grained object perception, inter-object reasoning, and scene-level interpretation and planning. Its underlying data are drawn from real city-scale point clouds with instance-level masks and landmark annotations, then transformed into a multimodal supervision format combining text, 2D imagery, and 3D-derived structured attributes (Chen et al., 24 Mar 2026).

1. Positioning and rationale

3DCity-LLM-1.2M is positioned as a response to a gap in prior 3D vision-language resources for urban environments (Chen et al., 24 Mar 2026). The stated motivation is that many earlier datasets are 2D-only and therefore do not expose the 3D spatial structure required for city-scale reasoning. Existing 3D urban datasets are described as often being limited to single tasks such as grounding or localization, centered on short-answer question answering rather than long-form reasoning, or too small and narrow to support LLM-scale instruction tuning.

Within that framing, the dataset is presented as jointly providing three properties that prior city-scale benchmarks fail to combine: large scale, 3D numerical grounding, and diverse user-oriented contextual simulation (Chen et al., 24 Mar 2026). According to the comparison reported in Table 1 of the paper, 3DCity-LLM-1.2M combines text, 2D, and 3D modalities; numerical information such as coordinates, distances, and angles; contextual simulation; approximately 1.2 million samples; and multiple tasks rather than a single narrowly defined benchmark.

A central design choice is that the dataset is not limited to object-centric supervision. Its task taxonomy spans fine-grained object analysis, relational computation, and scene planning, which places it closer to an instruction-style multimodal corpus than to a conventional urban QA benchmark. The paper also explicitly emphasizes its long-form character: the average question length is 13.49 words and 17.29 tokens, and the average answer length is 39.47 words and 49.44 tokens, in contrast to prior city-scale datasets such as NuScenes-QA and City-3DQA, whose reported average answer lengths are 1.04 and 1.80 words respectively (Chen et al., 24 Mar 2026).

This suggests that the dataset was designed not merely to scale sample count, but to shift the supervision target from short classification-style answers toward open-ended, explanatory, instruction-following responses grounded in urban geometry.

2. Source data, geographic scope, and multimodal representation

The dataset is built from three public city-scale 3D datasets: SensatUrban, UrbanBIS, and City-BIS (Chen et al., 24 Mar 2026). The geographic scope stated in the paper is:

Source dataset	Locations
SensatUrban	Birmingham and Cambridge, UK
UrbanBIS	Qingdao, Lihu, Longhua, Yuehai, and Wuhu, China
City-BIS	Heidelberg, Germany

The raw sources are city-scale point clouds with instance-level masks and landmark annotations. These are converted into a multimodal package used for data generation, consisting of 3D point clouds, 2D Bird’s-Eye View images, object-centric RGB crops, global scene views, and structured text descriptions containing geometry, semantics, and explicit 3D numeric values (Chen et al., 24 Mar 2026).

The paper describes this multimodal representation as essential because the VLM used for generation can directly process only text and images. Accordingly, raw 3D information is serialized into structured textual templates that include landmark names, coordinates, and pairwise distances. An example given in the paper is: “News Center (Building, located at [21.2, 417.3, 36.9]m) is approximately 45.2 meters from the parking lot (Parking Lot, located at [54.1, 448.9, 5.23]m), located to the southwest of the parking lot.” This mechanism makes absolute coordinates, metric distances, and relative direction explicit in the supervision rather than leaving them entirely implicit in geometry (Chen et al., 24 Mar 2026).

The resulting dataset is therefore grounded in 3D urban point-cloud scenes, but operationally expressed through a combined text, 2D image, and 3D-derived metadata representation. This differs from purely geometry-centric urban datasets such as TrueCity, which emphasizes synchronized real and simulated point clouds and semantic 3D city models for segmentation-oriented domain shift analysis rather than language supervision (Nguyen et al., 10 Nov 2025).

3. Task taxonomy and statistical structure

The paper defines seven task categories in a hierarchical taxonomy covering object-level, relationship-level, and scene-level reasoning (Chen et al., 24 Mar 2026). The internal dataset statistics reported in Table 3 are as follows:

Task category	Samples (proportion)	Average question / answer length
Object Caption	350k (28.3%)	10.32 / 60.46 words
Object Localization	94k (7.6%)	16.48 / 50.64 words
Object Analysis	470k (37.9%)	11.25 / 23.42 words
Relationship Computation	56k (4.6%)	25.45 / 40.62 words
Scene Caption	160k (12.9%)	9.73 / 32.40 words
Scene Analysis	55k (4.5%)	17.86 / 31.24 words
Scene Planning	52k (4.2%)	41.14 / 52.53 words

The largest category is Object Analysis at 470k samples, or 37.9% of the dataset, followed by Object Caption at 350k samples, or 28.3% (Chen et al., 24 Mar 2026). The smallest categories are Scene Planning, Scene Analysis, and Relationship Computation, each accounting for roughly four to five percent of the corpus.

Each category is associated with a distinct supervision objective. Object Caption requires textual description of an individual urban object. Object Localization requires retrieval of localization information such as coordinates or segmentation mask from language. Object Analysis targets deeper interpretation of physical characteristics, functional role, and affordances. Relationship Computation requires inference of quantitative spatial relationships and topological dependencies among objects. Scene Caption summarizes an entire city scene. Scene Analysis combines object-level, relational, and contextual evidence for high-level interpretation. Scene Planning targets goal-oriented reasoning and decision-making such as accessibility planning, routing, or intervention suggestion (Chen et al., 24 Mar 2026).

The reported token statistics also indicate nonuniform linguistic complexity. Relationship Computation is the most token-heavy category on the input side, with 53.83 average question tokens, while Scene Planning has the longest questions in words on average at 41.14 words (Chen et al., 24 Mar 2026). This distribution is consistent with the paper’s characterization of the dataset as long-form and instruction-oriented rather than uniformly templated.

The paper does not provide train, validation, and test split sizes for the full dataset in the supplied description, although it mentions a validation set in the quality-control section. A plausible implication is that operational split details may need to be obtained from the repository rather than the paper text alone.

4. Construction pipeline and annotation methodology

The dataset construction pipeline has two stages: city scene attribute extraction and VLM-driven QA pair generation (Chen et al., 24 Mar 2026).

In the first stage, the input point clouds from SensatUrban, UrbanBIS, and City-BIS are used together with instance-level masks and landmark annotations to build a city scene graph. In that graph, nodes correspond to objects and edges correspond to relations such as adjacency, containment, and orientation. The point clouds are also projected into 2D BEV images, with each object highlighted using a unique identifier. Because the generation model directly processes only text and images, the raw 3D information is transformed into object-centric RGB crops, global scene RGB views, and structured textual templates encoding geometry and semantics (Chen et al., 24 Mar 2026).

In the second stage, an advanced VLM, with ChatGPT-5 mentioned as an example, is prompted to generate QA pairs for multiple tasks (Chen et al., 24 Mar 2026). The paper emphasizes that this process is constrained rather than unconstrained free generation. Several generation rules are described.

First, diversity improvement is enforced by asking the model to produce multiple phrasings with the same semantics. The paper gives examples such as “Where is the nearest hospital?”, “Which location in the city corresponds to the closest hospital?”, and “Can you identify the hospital closest to this current place?” (Chen et al., 24 Mar 2026). This is intended to reduce rigidity associated with template-heavy datasets.

Second, a truthfulness requirement constrains answers to remain grounded in the provided scene attributes, including structured text with landmarks, attributes, and positions, as well as RGB image views. The model is instructed not to invent unsupported information (Chen et al., 24 Mar 2026).

Third, every item follows a fixed schema in which the question begins with <Question> and ends with </Question>, while the answer begins with <Answer> and ends with </Answer> (Chen et al., 24 Mar 2026). This yields an explicit instruction-response format suitable for supervised fine-tuning.

Fourth, the paper introduces contextual simulation through persona-conditioned prompting. Personas include tourist, government official, and company staff, each associated with different linguistic style, reasoning depth, and priorities (Chen et al., 24 Mar 2026). These persona conditions are realized through template-based learning with few-shot examples demonstrating expected tone, vocabulary, and reasoning style.

The resulting annotations are best characterized, in the paper’s own terms, as produced through an automated pipeline with VLM-generated QA pairs grounded in structured urban attributes (Chen et al., 24 Mar 2026). They are therefore synthetic or semi-automatic in language generation, but tied to real city data and structured scene evidence rather than being manually authored at scale.

5. Quality control, evaluation role, and benchmark protocol

The paper repeatedly describes 3DCity-LLM-1.2M as strictly quality-controlled (Chen et al., 24 Mar 2026). Its quality-control procedure is an automated cross-checking process on the validation set using multiple VLMs: ChatGPT-5, Gemini 2.5, and Claude-3.5-Sonnet. These evaluators are asked to identify residual template artifacts, privacy risk, ambiguous or ill-posed questions, uninformative, illogical, or overly short answers, and inconsistencies between answers and provided scene attributes (Chen et al., 24 Mar 2026). The stated effect of this procedure is the removal of hallucinations, privacy-sensitive content, and unclear language.

At the same time, the paper does not specify exact acceptance or rejection thresholds, inter-model voting logic, the deduplication algorithm, or quantitative retention and filtering rates (Chen et al., 24 Mar 2026). This incompleteness is important for reproducibility and downstream auditing. It also means that the phrase “strictly quality-controlled” is supported as a stated characterization, but the operational definition is only partially disclosed in the available text.

3DCity-LLM-1.2M also functions as an evaluation benchmark in the paper, together with City-3DQA (Chen et al., 24 Mar 2026). Because the dataset contains long-form outputs, the authors argue that standard text-overlap metrics are insufficient by themselves. The reported evaluation protocol therefore combines BLEU, ROUGE-L, and METEOR with two LLM-based semantic dimensions: Logicality and Reliability (Chen et al., 24 Mar 2026).

Logicality measures internal coherence, reasoning validity, non-contradiction, and whether conclusions follow from premises. Reliability measures factual correctness and alignment with scene evidence or ground truth, penalizing hallucinations and unsupported claims. These scores are assigned by three independent LLM evaluators—ChatGPT-5, Qwen3-VL Plus, and DeepSeek-V3—each of which sees the model-generated answer, the ground-truth answer, and the relevant 3D scene evidence without knowing which model produced the answer. Each evaluator assigns a score from 0 to 10 and provides a concise justification, and the final score is the average across evaluators (Chen et al., 24 Mar 2026).

The task-level averages reported in Table 10 show substantial variation across categories. For example, Object Analysis has average Logicality 8.09 and Reliability 6.95, whereas Relationship Computation has average Logicality 5.13 and Reliability 4.36 (Chen et al., 24 Mar 2026). The paper also reports a correlation heatmap among evaluator scores and claims moderate inter-evaluator correlations mostly in the 0.5 to 0.8 range.

6. Relationship to the 3DCity-LLM framework and technical consumption

The dataset is structurally aligned with the three-branch representation of the 3DCity-LLM model (Chen et al., 24 Mar 2026). Object-level tasks—Object Caption, Object Localization, and Object Analysis—align with the target object branch. Relationship Computation aligns with the inter-object relationship branch. Scene Caption, Scene Analysis, and Scene Planning align with the global scene branch.

The paper further describes a two-stage training strategy using 3DCity-LLM-1.2M (Chen et al., 24 Mar 2026). In the first stage, the model is trained on simple caption tasks to establish alignment across text, 2D images, and 3D point clouds. In the second stage, it is fine-tuned on high-level analysis and planning tasks, including Object Analysis, Relationship Computation, Scene Analysis, and Scene Planning, to enhance reasoning capacity for urban understanding and decision-making.

The model’s instruction parser determines which embeddings are active for a given task. For object-level and relationship-level tasks, one or more target objects are selected and the model uses object embedding $\mathbf{E}_o$, relationship embedding $\mathbf{E}_r$, and scene embedding $\mathbf{E}_s$. For scene-level tasks, no target object is selected, and the model uses scene embedding $\mathbf{E}_s$ while setting object and relationship embeddings to zero vectors (Chen et al., 24 Mar 2026). This indicates that the dataset is not merely a heterogeneous collection of tasks, but an explicitly structured supervision source matched to a hierarchical model architecture.

The paper also specifies the textual input representation. The text query $T$ is tokenized into a text feature vector $\mathbf{E}_{T} \in \mathbb{R}^{l \times d}$, where $l$ is a predefined sentence length and $d = 1024$ (Chen et al., 24 Mar 2026). The implementation uses an overall training sequence length of 512 tokens, although this is described as a model training detail rather than a dataset serialization limit.

Several formulas in the paper clarify how the dataset is consumed. The object embedding is defined as

$\mathbf{E}_{o} = \mathrm{Proj}_o([\mathbf{f}_{\text{v};\mathbf{f}_{\text{s};\mathbf{f}_{\text{l}]).$

The relationship attention weight is defined as

$\alpha_k = \frac{\exp\!\big(\mathbf{f}^{(t)}_{\text{s} \cdot [\mathbf{f}^{(k)}_{\text{s} + \phi(\Delta \mathbf{p}^{(k)})]\big)} {\sum_{j=1}^{K}\exp\!\big(\mathbf{f}^{(t)}_{\text{s} \cdot [\mathbf{f}^{(j)}_{\text{s} + \phi(\Delta \mathbf{p}^{(j)})]\big)}.$

The relationship embedding is

$\mathbf{E}_r$0

The scene embedding is

$\mathbf{E}_r$1

And the training loss is given as

$\mathbf{E}_r$2

These formulas are not dataset-generation equations, but they show how the dataset’s object, relational, and scene-level supervision interfaces with the model’s multimodal encoding and autoregressive training regime (Chen et al., 24 Mar 2026).

7. Limitations, caveats, and research significance

Several dataset-related caveats are explicit or implicit in the paper (Chen et al., 24 Mar 2026). First, although the supervision is grounded in real urban 3D data, the QA annotations are largely VLM-generated. This introduces the possibility of model-induced stylistic biases even when lexical and syntactic diversity is improved through paraphrastic generation. Second, the paper explicitly mentions filtering for “residual template artifacts,” which indicates that prompt-template traces can persist in the generated data. Third, operational details are incompletely disclosed: exact train, validation, and test partition sizes, acceptance thresholds in quality control, rejection rates, deduplication procedures, and per-city sample counts are not reported in the supplied text.

Fourth, the geographic scope, while multinational, remains limited to a finite set of cities in the United Kingdom, China, and Germany (Chen et al., 24 Mar 2026). This suggests that urban forms, architectural styles, and planning conventions represented in the dataset may not cover the full diversity of global cities. Fifth, because the dataset inherits instance masks and landmark annotations from existing source datasets, annotation errors or category biases in those resources may propagate into downstream supervision. Sixth, although explicit numeric grounding is a core strength, part of the 3D supervision is mediated through text serialization rather than exclusively through native geometric structures.

These caveats are relevant when comparing 3DCity-LLM-1.2M with geometry-centered urban benchmarks such as TrueCity (Nguyen et al., 10 Nov 2025). TrueCity is narrower in scope, focusing on synchronized real and simulated point clouds for semantic segmentation in a single urban corridor, but it emphasizes manually labeled real-world data, standards-aligned class definitions, and tightly controlled sim-to-real analysis. By contrast, 3DCity-LLM-1.2M prioritizes scale, multimodal integration, long-form instruction supervision, and language-enabled urban reasoning (Chen et al., 24 Mar 2026, Nguyen et al., 10 Nov 2025). The two datasets therefore occupy different positions within urban AI research: one as a broad multimodal instruction corpus for city-scale perception and understanding, the other as a controlled geometry benchmark for cross-domain 3D scene understanding.

The practical significance of 3DCity-LLM-1.2M lies in the specific combination of real 3D urban geometry, explicit numerical information, multimodal representations, and persona-conditioned query simulation (Chen et al., 24 Mar 2026). The paper’s release statement says that the source code and dataset are available at the SYSU-3DSTAILab repository. The provided text does not specify a dataset license. A plausible implication is that researchers intending to use the corpus operationally must consult the repository for packaging, licensing, and split details not fully described in the paper.

Within the literature on multimodal urban understanding, 3DCity-LLM-1.2M is therefore best understood as a large-scale, instruction-style, city-scale 3D vision-language dataset whose principal novelty lies not only in sample count, but in the coupling of explicit 3D numerical grounding with long-form, context-sensitive supervision for object perception, spatial reasoning, scene interpretation, and planning (Chen et al., 24 Mar 2026).