3DCity-LLM: City-Scale Multimodal Perception

• 3DCity-LLM is a city-scale multimodal framework that integrates hierarchical encoders with instruction-tuned language models for urban scene understanding.
• It employs a coarse-to-fine three-branch encoder, fusing 3D point clouds, RGB projections, and textual landmark cues to capture object, relational, and global context.
• The framework is trained on an extensive 1.2M-sample dataset across seven urban task categories, achieving state-of-the-art results in object, relationship, and scene tasks.

Searching arXiv for the primary paper and related city-scale 3D language-model work. 3DCity-LLM denotes a multimodality LLM framework for 3D city-scale perception and understanding. It was introduced as a unified system for city-scale vision-language reasoning that combines a coarse-to-fine three-branch encoder with an instruction-tuned language backbone, and it is trained on 3DCity-LLM-1.2M, a dataset of approximately 1.2 million samples spanning seven urban task categories. The framework is explicitly designed for urban environments in which thousands of entities, long-range spatial dependencies, landmark semantics, and quantitative geometric relations must be handled jointly rather than as isolated object-centric predictions (Chen et al., 24 Mar 2026).

1. Research setting and problem formulation

3DCity-LLM is motivated by the mismatch between conventional multimodal LLMs and the structure of urban scenes. The paper characterizes city-scale environments as difficult because they involve large spatial extent, high entity density, geometry- and topology-aware relations, and holistic scene semantics. In this setting, queries such as identifying the hospital closest to a railway station and locating its emergency department require the model to recognize categories, read precise coordinates, compute distances and directions, and place those facts within a global layout (Chen et al., 24 Mar 2026).

The framework therefore departs from object-centric or indoor formulations. It does not assume that urban reasoning can be reduced to a single local crop or a short-form question-answering setup. Instead, it organizes perception into three levels: target object, inter-object relationship, and global scene. This suggests a hierarchical view of city understanding in which fine appearance, local relational structure, and large-scale contextual layout are treated as distinct but composable sources of evidence.

The paper also positions 3DCity-LLM against a broader methodological background. Earlier city-oriented systems capture only parts of the problem: LiDAR-LLM reformulates sparse outdoor LiDAR cognition as a language modeling problem and supports captioning, grounding, and question answering over urban scenes, but it is centered on LiDAR plus a View-Aware Transformer and a three-stage alignment curriculum rather than a unified city-scale multimodal hierarchy (Yang et al., 2023). SpatialLLM, by contrast, compiles maps, imagery, and point clouds into a structured textual scene description and performs zero-shot urban reasoning without training, emphasizing prompt construction rather than instruction-tuned multimodal fusion (Chen et al., 19 May 2025). 3DCity-LLM occupies a different position: it is a trainable city-scale multimodality LLM with explicit 3D numerics, structured branch design, and a dedicated evaluation protocol (Chen et al., 24 Mar 2026).

2. Architectural design

The architecture is organized around a coarse-to-fine three-branch encoder fused into an instruction-tuned LLM. Its inputs include city-scale point clouds from SensatUrban, UrbanBIS, and City-BIS; top-view RGB projections for both object-centric and global views; and textual attributes such as landmark names, coordinates, distances, and orientation descriptors. Instance masks are obtained with SoftGroup, top-view RGB crops are processed with CLIP, object-level 3D points are encoded with Uni3D, landmark names are encoded with BERT, and the language backbone is LLaVA-v1.5-7B built on LLaMA (Chen et al., 24 Mar 2026).

The target object branch captures object appearance, geometry, and semantic label or landmark identity. It concatenates a local top-view visual token sequence from CLIP, a geometry-aware token from Uni3D, and a landmark text token from BERT when present; the result is projected into the LLM embedding space by a learned projector $Proj_o$. The inter-object relationship branch retrieves $K$ nearest neighbors around the target, encodes each neighbor’s points with Uni3D, computes relative offsets $\Delta p(k) = p(o_k) - p(o_t)$, and injects them through a learnable spatial function $\phi(\cdot)$. Its attention-style weighting is defined as

$$\alpha_k = \frac{\exp\left(f_s^{(t)} \cdot [f_s^{(k)} + \phi(\Delta p(k))]\right)}{\sum_j \exp\left(f_s^{(t)} \cdot [f_s^{(j)} + \phi(\Delta p(j))]\right)}.$$

The weighted geometry tokens and optional landmark tokens are projected by $Proj_r$. The global scene branch encodes the citywide top-view RGB map with CLIP, combines this with global landmark tokens from BERT, and projects the result by $Proj_s$ (Chen et al., 24 Mar 2026).

These branch outputs are aligned to the LLM space of dimension $d_{LLM}=4096$:

$$E_o = Proj_o([f_v; f_s; f_l]), \quad E_r = Proj_r([F_g; F_l]), \quad E_s = Proj_s([F_v^{Sce}; F_l^{Sce}]).$$

The full input to the LLM is

$E_{in} = [E_T; E_o; E_r; E_s].$

Instruction-driven activation determines which branches are used. For scene-level tasks, only $K$0 is active; for object- and relationship-level tasks, $K$1, $K$2, and $K$3 are all used. This branch-selective conditioning is central to the framework’s claim of task generality without architectural changes (Chen et al., 24 Mar 2026).

Spatial reasoning is grounded in explicit primitives. The paper states distances as $K$4, heading as $K$5, and rigid transformation as $K$6 with $K$7. Because relative positions are computed per target while global top-view maps preserve scene context, the design combines local normalization with citywide reference structure (Chen et al., 24 Mar 2026).

3. 3DCity-LLM-1.2M dataset

The 3DCity-LLM-1.2M dataset is a defining component of the framework. It contains approximately 1.2 million samples built from SensatUrban, UrbanBIS, and City-BIS, covering cities in Europe and China, and it spans seven task categories: Object Caption, Object Localization, Object Analysis, Relationship Computation, Scene Caption, Scene Analysis, and Scene Planning (Chen et al., 24 Mar 2026).

The task distribution is explicitly reported as follows: Object Caption 350k, Object Localization 94k, Object Analysis 470k, Relationship Computation 56k, Scene Caption 160k, Scene Analysis 55k, and Scene Planning 52k. The answers are long-form rather than merely categorical; the paper notes, for example, that object analysis has an average answer length of approximately 23 words, object caption approximately 60 words, and scene planning approximately 53 words (Chen et al., 24 Mar 2026).

A central feature of the dataset is the incorporation of explicit 3D numerics. Structured attributes serialize coordinates, pairwise distances, and directional cues, enabling responses such as “approximately 45 m southwest.” The dataset also introduces role-conditioned simulations with personas such as tourist, government official, and company staff, which diversify linguistic style and reasoning depth. Automated generation proceeds by extracting scene attributes, serializing them into structured texts, and prompting vision-LLMs to synthesize question-answer pairs with format guards and truthfulness constraints. Validation then uses multiple LLM evaluators—ChatGPT-5, Gemini 2.5, and Claude-3.5 Sonnet—to identify template artifacts, privacy risks, ambiguity, short or illogical answers, and inconsistencies with the structured attributes (Chen et al., 24 Mar 2026).

This dataset construction differs from earlier urban reasoning corpora in both scale and annotation style. LiDAR-LLM created large LiDAR-text corpora through three-stage generation and alignment for captioning and grounding, but its nu-Caption and nu-Grounding datasets are tied to autonomous-driving LiDAR rather than city-scale multimodal urban reasoning (Yang et al., 2023). HoliCity, in turn, provides 6,300 high-resolution panoramas aligned with CAD over more than $K$8, supplying structurally rich supervision for city geometry, but not the long-form instruction-following and persona-driven QA structure introduced in 3DCity-LLM-1.2M (Zhou et al., 2020).

4. Training procedure and evaluation protocol

The training objective is token-level cross-entropy for instruction-following question answering. For an input-target pair $K$9, the paper defines

$\Delta p(k) = p(o_k) - p(o_t)$0

The paper also gives a more general form,

$\Delta p(k) = p(o_k) - p(o_t)$1

while stating that $\Delta p(k) = p(o_k) - p(o_t)$2 is the main term in the reported experiments (Chen et al., 24 Mar 2026).

Implementation details are concrete. The model uses LLaVA-v1.5-7B with trainable projectors and LoRA adapters on attention and FFN layers. Training is performed in PyTorch with bf16 on a single NVIDIA A100, using DeepSpeed ZeRO-2 for memory. The reported schedule is two-stage instruction tuning: Stage 1 on captioning for approximately 10 hours, then Stage 2 on analysis and planning for approximately 24 hours. Hyperparameters include LoRA rank 8, dropout 0.05, maximum sequence length 512, batch size 8, gradient accumulation 10, and AdamW with learning rate $\Delta p(k) = p(o_k) - p(o_t)$3, followed by a 10-step warm-up and decay (Chen et al., 24 Mar 2026).

The evaluation protocol is one of the framework’s distinctive claims. Conventional text-similarity metrics are retained—BLEU-4, ROUGE-L, and METEOR—but they are supplemented by LLM-based semantic assessment under two criteria: Logicality and Reliability, each scored from 0 to 10. Three independent LLMs—ChatGPT-5, Qwen3-VL Plus, and DeepSeek-V3—grade answers using the prediction, reference, and structured scene evidence, and they are required to provide justifications. The paper reports moderate inter-evaluator correlations of approximately $\Delta p(k) = p(o_k) - p(o_t)$4 to $\Delta p(k) = p(o_k) - p(o_t)$5, presenting this as evidence that the protocol reduces evaluator bias while preserving traceability (Chen et al., 24 Mar 2026).

This evaluation design addresses a known problem in urban reasoning systems: correct long-form answers can vary lexically while remaining semantically faithful. SpatialLLM had already argued that multi-field knowledge, context length, and reasoning ability strongly influence urban analysis, and it evaluated zero-shot scene-description prompting with multiple-choice spatial perception tasks (Chen et al., 19 May 2025). 3DCity-LLM extends that concern into an instruction-tuned benchmark regime for open-ended urban reasoning, where textual overlap alone is an incomplete measure (Chen et al., 24 Mar 2026).

5. Results, ablations, and operational characteristics

On 3DCity-LLM-1.2M, the framework reports the best overall results among the compared 7B multimodal models. For object-level tasks, it achieves BLEU-4 30.64, ROUGE-L 42.81, METEOR 48.89, Logicality 7.33, and Reliability 6.02. For relationship-level tasks, it reports BLEU-4 20.98, ROUGE-L 34.61, METEOR 42.09, Logicality 5.13, and Reliability 4.36. For scene-level tasks, it reports BLEU-4 20.11, ROUGE-L 29.87, METEOR 37.84, Logicality 7.50, and Reliability 6.16 (Chen et al., 24 Mar 2026).

The subtask breakdown is similarly specific. The model attains BLEU-4 13.80, ROUGE-L 23.24, and METEOR 32.25 on object caption; METEOR 39.41 with Logicality 6.28 and Reliability 4.52 on object localization; BLEU-4 39.22 and METEOR 57.02 on object analysis; BLEU-4 25.01 and METEOR 35.08 on scene caption; BLEU-4 17.42 and METEOR 36.60 on scene analysis; and the highest Logicality 7.51 and Reliability 6.12 on scene planning (Chen et al., 24 Mar 2026).

On City-3DQA, the framework reports sentence-wise accuracy 68.55, including 82.41 single-hop and 57.75 multi-hop. In city-wise evaluation, it reaches 79.10 on single-hop and 64.49 overall, which is second to City-VLM’s 64.70 overall. The paper therefore presents 3DCity-LLM as strongest on sentence-wise and multi-hop behavior while acknowledging a marginal deficit in one city-wise aggregate (Chen et al., 24 Mar 2026).

Ablations support the three-branch design. In the object branch, local view alone gives BLEU-4 29.61 and METEOR 45.94; adding shape helps slightly; adding landmark information reaches BLEU-4 30.64 and METEOR 48.89, while Reliability improves from 5.64 to 6.02. In the relationship branch, geometry only yields BLEU-4 18.24 and Reliability 4.04; adding landmark tokens raises these to 20.98 and 4.36. In the scene branch, global view alone gives BLEU-4 19.72 and METEOR 35.82, and adding landmarks increases these to 20.11 and 37.84 while also improving Logicality and Reliability (Chen et al., 24 Mar 2026).

The paper also reports average per-query inference time on a single A100: 7.40 s for object-level tasks, 13.69 s for relationship-level tasks, and 14.54 s for scene-level tasks. The longer latency of relationship- and scene-level queries is attributed to extra neighbor encoding, spatial computations, global context, and longer reasoning. This suggests that the system’s computational cost is strongly tied to how much explicit relational and scene context is activated.

6. Relation to adjacent research, misconceptions, and open problems

3DCity-LLM belongs to a rapidly forming research area on city-scale language-conditioned perception, but it should not be conflated with several neighboring paradigms. It is not a pure text-to-text urban analytics system of the kind exemplified by SpatialLLM, which uses structured scene descriptions and zero-shot prompting without model training (Chen et al., 19 May 2025). It is not a city-scale visual programming system in the style of GeoProg3D, where an LLM composes geographic vision APIs over a geography-aware 3D language field for grounding, comparison, counting, and measurement (Yasuki et al., 29 Jun 2025). Nor is it a generative urban modeling system such as CityDreamer, which focuses on compositional generation of unbounded 3D cities using BEV layouts, neural fields, and localized editing rather than multimodal urban QA (Xie et al., 2023).

A common misconception is that city-scale multimodal reasoning can be achieved by simply scaling up indoor or object-centric MLLMs. The paper argues against this directly by emphasizing city scenes with thousands of entities, heterogeneous geometry and semantics, and long-range relations that require target, relational, and global context simultaneously (Chen et al., 24 Mar 2026). Another misconception is that high performance on short-form urban QA is sufficient. The introduction of Logicality and Reliability, together with persona-driven long-form annotations, indicates that the authors view factual faithfulness and reasoning coherence as distinct evaluation targets.

The framework nonetheless retains important limitations. Because of compute constraints, experiments use a 7B LLM backbone with LoRA; the paper states that larger backbones such as 13B or 34B could improve long-range dependencies and multi-hop reasoning. Relationship-level logicality and reliability remain lower than object- and scene-level values. The authors also note that very dense or occluded areas and extremely long-range dependencies may stress neighbor selection and memory. Performance depends on landmark availability, as shown by branch ablations, and despite depersonalized data plus automated privacy checks, the paper states that deployment in real cities must respect data governance and avoid reinforcing inequities in urban planning recommendations (Chen et al., 24 Mar 2026).

In the broader city-scale ecosystem, these limitations point toward integration opportunities rather than contradictions. HoliCity provides accurately aligned panorama-CAD supervision for structural reasoning over more than $\Delta p(k) = p(o_k) - p(o_t)$6 of London (Zhou et al., 2020). Digital-twin work on automatic mesh generation offers simulation-grade LoD1.2 city meshes derived from cadastral footprints and aerial point clouds for large-scale modeling workflows (Naserentin et al., 2022). GeoProg3D demonstrates that tool-compositional geographic reasoning over high-fidelity 3D scenes is feasible at city scale (Yasuki et al., 29 Jun 2025). A plausible implication is that future urban multimodality LLMs may combine 3DCity-LLM’s hierarchical encoder and instruction tuning with stronger 3D world models, explicit geometric tool use, and richer digital-twin substrates.

In that sense, 3DCity-LLM is best understood as a city-scale multimodality LLM benchmark and architecture rather than a complete urban intelligence stack. Its principal contribution is the joint establishment of a hierarchical perception design, a large instruction-tuning corpus with explicit 3D numerics, and a semantics-aware evaluation protocol for long-form urban reasoning (Chen et al., 24 Mar 2026).