CitySeg: City-Scale 3D Segmentation

• The paper presents CitySeg as a foundation model that unifies multi-dataset training with density normalization and hierarchical text embeddings for open-vocabulary segmentation.
• CitySeg employs a local-global cross-attention mechanism and multi-resolution grid sampling to effectively handle scale, sparse geometry, and label mismatches in urban point clouds.
• Evaluations across nine benchmarks demonstrate state-of-the-art performance with improved OA and mIoU, supporting shared-weight inference and fine-tuning for new datasets.

Searching arXiv for CitySeg and closely related papers on city-scale segmentation and urban segregation. CitySeg is a foundation model for city-scale point cloud semantic segmentation in UAV scenarios that incorporates text modality to achieve open vocabulary segmentation and zero-shot inference without relying on visual information (Xu et al., 13 Aug 2025). It addresses the joint-training regime in which multiple city-scale datasets must be combined despite divergent point-cloud distributions and non-uniform annotations, two factors that otherwise induce negative transfer and poor cross-domain generalization. In this formulation, CitySeg is not a conventional closed-set, dataset-specific segmenter, but a multi-domain framework that couples point-cloud encoding, hierarchy-aware text semantics, and graph-based label unification so that a single model can support shared-weight inference across datasets, fine-tuning on new datasets, and category extension through text and graph structure (Xu et al., 13 Aug 2025).

1. Problem domain and scope

CitySeg targets 3D semantic segmentation of very large urban point clouds, often acquired by LiDAR or multi-view stereo, where the task is to assign a semantic label to every point in a city-scale scene (Xu et al., 13 Aug 2025). The intended deployment setting is UAV perception, and the paper emphasizes four technical difficulties: scale and memory constraints, sparse and non-uniform geometry, the need for global context, and cross-dataset label mismatch. In city scenes, local geometry is frequently ambiguous: a flat planar patch may correspond to ground or roof, while a vertical surface may correspond to building wall or bridge structure. This motivates a representation that preserves local detail while injecting scene-level context.

The paper frames the main failure mode of prior multi-dataset training as the conjunction of two heterogeneities. First, different UAV sensors and reconstruction pipelines produce point clouds with very different densities, scales, and sampling patterns. Second, different datasets define semantic categories at different granularities; one dataset may label “car” as a broad mobile vehicle category, while another separates “car” and “truck.” CitySeg is therefore organized around the claim that generalization requires simultaneous normalization of geometric distributions and reconciliation of semantic taxonomies rather than only a stronger point backbone (Xu et al., 13 Aug 2025).

This also explains why the model is described as a foundation model. The term refers here to training on multiple city-scale datasets, using a frozen pre-trained text encoder, aligning point embeddings to hierarchical text embeddings, and supporting zero-shot and open-vocabulary generalization instead of only closed-set prediction on a single benchmark (Xu et al., 13 Aug 2025).

2. Data preprocessing and local-global geometric encoding

A first component of CitySeg is a set of preprocessing rules intended to mitigate non-uniform multi-domain point-cloud distributions (Xu et al., 13 Aug 2025). Missing attributes are filled with zeros; for example, LiDAR datasets without color have absent RGB channels set to zero. Grid sampling is then used to unify point density across datasets, reduce memory consumption, and normalize spatial granularity.

The sampling procedure is explicitly multi-resolution. For each batch, the model samples local points $\mathbf{P}_{loc}$ with size $M$ and a larger global neighborhood $\mathbf{P}_{glo}$ around those local points with size $10M$. The paper states a local grid size of $0.2\,\mathrm{m}$ and a global grid size of $1\,\mathrm{m}$. After sampling, both local and global point sets are serialized using a space-filling curve for structured processing (Xu et al., 13 Aug 2025). This suggests that CitySeg treats density normalization not as a peripheral engineering step but as part of the semantic modeling pipeline itself.

The point network comprises an encoder $f_{pe}$, a local-global cross-attention module, and a decoder $f_{pd}$. Feature extraction is written as

$$\mathbf{F}_{loc} = f_{pe}(\mathcal{G}_{loc}(\mathbf{P}_{loc})), \quad \mathbf{F}_{glo} = f_{pe}(\mathcal{G}_{glo}(\mathbf{P}_{glo})).$$

The local branch serves as query and the global branch provides key and value:

$$\mathbf{F}_{attn} = \mathrm{softmax}(Q_{loc}K_{glo}^{T})V_{glo}.$$

The output embedding is then

$M$0

The architectural role of this module is explicit in the paper: local features preserve fine details, global features add scene-level context, and the combined representation helps distinguish ambiguous structures in sparse or uneven city-scale point clouds (Xu et al., 13 Aug 2025). In ablation, adding local-global attention improves OA from $M$1 to $M$2 and mIoU from $M$3 to $M$4, which is the paper’s direct evidence that cross-scale context is central rather than auxiliary (Xu et al., 13 Aug 2025).

3. Hierarchical label unification and graph-encoded text semantics

A second major component is CitySeg’s hierarchical classification strategy, introduced to resolve semantic label discrepancies across datasets (Xu et al., 13 Aug 2025). Instead of forcing all labels into a flat joint ontology, the method constructs a hierarchical classification graph from dataset annotation rules. The graph is defined as

$M$5

where nodes are labels and edges encode parent-child relationships. The root node is a universal semantic class, each node has exactly one parent, every node has a unique path from the root, and depth is determined by annotation rules rather than manually fixed.

The distinction between base classes and subclasses is structurally important. Base classes are broad labels with no overlapping objects, while subclasses are finer labels that are subsets of base classes. The paper adds that the text strings are not changed; only their hierarchical level is determined by annotation policy (Xu et al., 13 Aug 2025). This means label reconciliation is performed through hierarchy assignment rather than lexical rewriting.

Text semantics enter through a frozen pre-trained text encoder $M$6 that produces initial text embeddings $M$7, followed by a graph encoder $M$8 that models hierarchical relations:

$M$9

The resulting hierarchical text embedding $\mathbf{P}_{glo}$0 is then used for classification by similarity with point embeddings (Xu et al., 13 Aug 2025). The graph encoder is described as capturing semantic content from text, hierarchical granularity, parent-child relationships, and sibling similarity and difference.

This hierarchy-aware text branch is the mechanism that supports open-vocabulary behavior. The text encoder provides semantic class descriptions and a shared embedding space for labels, and new categories can be added by inserting a new leaf node into the graph, computing its hierarchical embedding, and applying the same similarity-based classifier (Xu et al., 13 Aug 2025). The paper is explicit that this procedure enables category extension without redesigning the model.

4. Training objective and open-vocabulary inference

CitySeg uses a two-stage training strategy intended to stabilize optimization under multi-domain semantic inconsistency (Xu et al., 13 Aug 2025). In Stage 1, all labels are merged into five base categories. For a point $\mathbf{P}_{glo}$1, point embedding $\mathbf{P}_{glo}$2, and hierarchical text embedding $\mathbf{P}_{glo}$3, the classification probability is

$\mathbf{P}_{glo}$4

where $\mathbf{P}_{glo}$5 is cosine similarity and $\mathbf{P}_{glo}$6 is temperature. The loss in this stage is Cross-Entropy loss $\mathbf{P}_{glo}$7, with the stated purpose of learning broad urban semantics first and establishing a stable initialization for finer recognition (Xu et al., 13 Aug 2025).

In Stage 2, the model learns all subclasses and focuses on confusing sibling categories. The total loss is

$\mathbf{P}_{glo}$8

where $\mathbf{P}_{glo}$9 balances a hinge loss term. For point embedding $10M$0, ground-truth hierarchical text embedding $10M$1, and sibling set $10M$2, the hinge loss is

$10M$3

with $10M$4 denoting Euclidean distance and $10M$5 a margin (Xu et al., 13 Aug 2025). The intended effect is to pull a point embedding toward the correct label embedding while pushing it away from sibling categories by at least the margin.

The paper presents this hinge term as a remedy for fine-grained sibling collapse under ordinary cross-entropy. In ablation, adding hinge loss improves OA from $10M$6 to $10M$7 and mIoU from $10M$8 to $10M$9 (Xu et al., 13 Aug 2025). The open-vocabulary setting follows directly from the same similarity rule used during training: zero-shot inference is performed by encoding point embeddings, comparing them to hierarchical text embeddings, and predicting the most similar class. The model therefore performs open-vocabulary segmentation without relying on visual information, a property the paper identifies as especially relevant for LiDAR-only UAV systems (Xu et al., 13 Aug 2025).

5. Benchmarks, implementation, and empirical performance

CitySeg is jointly trained on nine city-scale benchmarks: SensatUrban, Campus3D, STPLS3D, Swiss3Dcities, Hessigheim 3D, SUM, DALES, DublinCity, and ECLAIR, with UrbanBIS used as a separate open-set test domain (Xu et al., 13 Aug 2025). The implementation uses Pointcept, a pre-trained PTv3 backbone, and a frozen CLIP text encoder. Training uses 8 NVIDIA A800 GPUs, batch size 16, 100 epochs, AdamW, learning rate $0.2\,\mathrm{m}$0, OneCycleLR, and inputs consisting of 3D coordinates plus color. The local and global point counts per batch are 65,536 and 655,360 respectively, and the hinge-loss weight is $0.2\,\mathrm{m}$1 (Xu et al., 13 Aug 2025).

Evaluation setting	Result
SensatUrban, PTv3	93.9 OA / 68.4 mIoU
SensatUrban, CitySeg	95.2 OA / 72.0 mIoU
UrbanBIS, PTv3 + PPT	53.7 OA / 42.6 mIoU
UrbanBIS, CitySeg-FM	89.7 OA / 65.1 mIoU
UrbanBIS, fully supervised CitySeg	92.0 OA / 70.4 mIoU

The paper distinguishes two evaluation modes: CitySeg-FM, which uses shared weights across datasets, and CitySeg-FT, which is fine-tuned on each dataset (Xu et al., 13 Aug 2025). Its overall claim is that CitySeg-FM already beats prior multi-dataset methods on most benchmarks, while CitySeg-FT achieves best results on all nine benchmarks. The most explicit per-dataset improvement reported in the data block is on SensatUrban, where PTv3 obtains $0.2\,\mathrm{m}$2 OA / $0.2\,\mathrm{m}$3 mIoU and CitySeg obtains $0.2\,\mathrm{m}$4 OA / $0.2\,\mathrm{m}$5 mIoU, corresponding to gains of $0.2\,\mathrm{m}$6 OA and $0.2\,\mathrm{m}$7 mIoU (Xu et al., 13 Aug 2025).

The UrbanBIS result is the strongest zero-shot demonstration. UrbanBIS is held out during training, and CitySeg-FM reaches $0.2\,\mathrm{m}$8 OA / $0.2\,\mathrm{m}$9 mIoU, compared with $1\,\mathrm{m}$0 OA / $1\,\mathrm{m}$1 mIoU for PTv3 + PPT (Xu et al., 13 Aug 2025). The paper also notes that this zero-shot model remains only $1\,\mathrm{m}$2 OA points below the fully supervised CitySeg upper bound on UrbanBIS, reported as $1\,\mathrm{m}$3 OA / $1\,\mathrm{m}$4 mIoU. This is the basis for the statement that, for the first time, CitySeg enables zero-shot generalization in city-scale point cloud scenarios without relying on visual information (Xu et al., 13 Aug 2025).

A plausible implication is that CitySeg’s empirical contribution lies less in a single architectural novelty than in the coordinated treatment of density normalization, cross-scale context, label hierarchy, and text alignment. The reported results are organized to support precisely that interpretation.

6. Terminology, adjacent usages, and conceptual context

The term “CitySeg” is not used uniformly across the literature, and distinguishing these usages is important. CitySeg in the strict sense refers to the 3D open-vocabulary semantic segmentation foundation model for city-scale point clouds described above (Xu et al., 13 Aug 2025). By contrast, CitySeg/MOP is a benchmark test suite that turns hardware-aware neural architecture search for real-time semantic segmentation into fifteen standard multi-objective optimization problems derived from the Cityscapes dataset; it is a benchmark for optimization over segmentation architectures, not a 3D segmentation model (Zhao et al., 2024).

Within semantic segmentation proper, CitySeg is also distinct from 2D urban-scene models such as SegRet and ProtoSeg. SegRet is an efficient encoder-decoder design for semantic segmentation that uses a Vision RetNet backbone and a lightweight residual decoder with zero-initialization, reporting Cityscapes results such as $1\,\mathrm{m}$5 mIoU under single-scale evaluation for its Tiny model with $1\,\mathrm{m}$6M parameters (Li et al., 19 Feb 2025). ProtoSeg, by contrast, is an interpretable semantic segmentation model based on class-specific prototypes and training-set patches, evaluated on Pascal VOC and Cityscapes, with Cityscapes mIoU of $1\,\mathrm{m}$7 on validation and $1\,\mathrm{m}$8 on test for the ImageNet-pretrained version (Sacha et al., 2023). These are image-based 2D approaches; CitySeg is a point-cloud foundation model for UAV-scale 3D scenes.

A further source of ambiguity is the broader “CitySeg-related” literature on urban segregation and city partitioning. MotifGPL studies urban social segregation through prototype-based graph structure extraction, motif distribution discovery, and urban graph reconstruction at the city-block level (He et al., 2024). Other work measures socio-spatial segregation through multilayer transport-network random walks (Neira et al., 2023), mobility-derived experienced segregation across rings and pockets in U.S. cities (Renninger et al., 2024), and imagery-based perception of exposure segregation through the VISAGE framework (Zhang et al., 20 Jun 2026). These are city-level segmentation or segregation analysis methods in the urban-science sense, not semantic segmentation models in computer vision.

This terminological divergence matters because CitySeg sits at the intersection of several neighboring traditions—city-scene parsing, foundation-model design, cross-dataset taxonomy alignment, and urban-scale spatial analysis—while belonging unambiguously to only one of them: city-scale 3D semantic segmentation for UAV perception (Xu et al., 13 Aug 2025).