---
title: 'CitySeg: City-Scale 3D Segmentation'
url: https://www.emergentmind.com/topics/cityseg
type: topic
---

# CitySeg: City-Scale 3D Segmentation

Searching arXiv for CitySeg and closely related papers on city-scale segmentation and urban segregation.
CitySeg is a foundation model for city-scale point cloud semantic segmentation in UAV scenarios that incorporates text modality to achieve open vocabulary segmentation and zero-shot inference without relying on visual information [2508.09470]. It addresses the joint-training regime in which multiple city-scale datasets must be combined despite divergent point-cloud distributions and non-uniform annotations, two factors that otherwise induce negative transfer and poor cross-domain generalization. In this formulation, CitySeg is not a conventional closed-set, dataset-specific segmenter, but a multi-domain framework that couples point-cloud encoding, hierarchy-aware text semantics, and graph-based label unification so that a single model can support shared-weight inference across datasets, fine-tuning on new datasets, and category extension through text and graph structure [2508.09470].

## 1. Problem domain and scope

CitySeg targets 3D semantic segmentation of very large urban point clouds, often acquired by LiDAR or multi-view stereo, where the task is to assign a semantic label to every point in a city-scale scene [2508.09470]. The intended deployment setting is UAV perception, and the paper emphasizes four technical difficulties: scale and memory constraints, sparse and non-uniform geometry, the need for global context, and cross-dataset label mismatch. In city scenes, local geometry is frequently ambiguous: a flat planar patch may correspond to ground or roof, while a vertical surface may correspond to building wall or bridge structure. This motivates a representation that preserves local detail while injecting scene-level context.

The paper frames the main failure mode of prior multi-dataset training as the conjunction of two heterogeneities. First, different UAV sensors and reconstruction pipelines produce point clouds with very different densities, scales, and sampling patterns. Second, different datasets define semantic categories at different granularities; one dataset may label “car” as a broad mobile vehicle category, while another separates “car” and “truck.” CitySeg is therefore organized around the claim that generalization requires simultaneous normalization of geometric distributions and reconciliation of semantic taxonomies rather than only a stronger point backbone [2508.09470].

This also explains why the model is described as a foundation model. The term refers here to training on multiple city-scale datasets, using a frozen pre-trained text encoder, aligning point embeddings to hierarchical text embeddings, and supporting zero-shot and open-vocabulary generalization instead of only closed-set prediction on a single benchmark [2508.09470].

## 2. Data preprocessing and local-global geometric encoding

A first component of CitySeg is a set of preprocessing rules intended to mitigate non-uniform multi-domain point-cloud distributions [2508.09470]. Missing attributes are filled with zeros; for example, LiDAR datasets without color have absent RGB channels set to zero. Grid sampling is then used to unify point density across datasets, reduce memory consumption, and normalize spatial granularity.

The sampling procedure is explicitly multi-resolution. For each batch, the model samples local points $\mathbf{P}_{loc}$ with size $M$ and a larger global neighborhood $\mathbf{P}_{glo}$ around those local points with size $10M$. The paper states a local grid size of $0.2\,\mathrm{m}$ and a global grid size of $1\,\mathrm{m}$. After sampling, both local and global point sets are serialized using a space-filling curve for structured processing [2508.09470]. This suggests that CitySeg treats density normalization not as a peripheral engineering step but as part of the semantic modeling pipeline itself.

The point network comprises an encoder $f_{pe}$, a local-global cross-attention module, and a decoder $f_{pd}$. Feature extraction is written as
$$
\mathbf{F}_{loc} = f_{pe}(\mathcal{G}_{loc}(\mathbf{P}_{loc})), \quad \mathbf{F}_{glo} = f_{pe}(\mathcal{G}_{glo}(\mathbf{P}_{glo})).
$$
The local branch serves as query and the global branch provides key and value:
$$
\mathbf{F}_{attn} = \mathrm{softmax}(Q_{loc}K_{glo}^{T})V_{glo}.
$$
The output embedding is then
$$
\mathbf{E}_p = f_{pd}(\mathrm{cat}(\mathbf{F}_{loc}, FFN(\mathbf{F}_{attn}))) \in \mathbb{R}^{M \times C}.
$$

The architectural role of this module is explicit in the paper: local features preserve fine details, global features add scene-level context, and the combined representation helps distinguish ambiguous structures in sparse or uneven city-scale point clouds [2508.09470]. In ablation, adding local-global attention improves OA from $89.8$ to $93.6$ and mIoU from $65.0$ to $67.8$, which is the paper’s direct evidence that cross-scale context is central rather than auxiliary [2508.09470].

## 3. Hierarchical label unification and graph-encoded text semantics

A second major component is CitySeg’s hierarchical classification strategy, introduced to resolve semantic label discrepancies across datasets [2508.09470]. Instead of forcing all labels into a flat joint ontology, the method constructs a hierarchical classification graph from dataset annotation rules. The graph is defined as
$$
H = (V, E),
$$
where nodes are labels and edges encode parent-child relationships. The root node is a universal semantic class, each node has exactly one parent, every node has a unique path from the root, and depth is determined by annotation rules rather than manually fixed.

The distinction between base classes and subclasses is structurally important. Base classes are broad labels with no overlapping objects, while subclasses are finer labels that are subsets of base classes. The paper adds that the text strings are not changed; only their hierarchical level is determined by annotation policy [2508.09470]. This means label reconciliation is performed through hierarchy assignment rather than lexical rewriting.

Text semantics enter through a frozen pre-trained text encoder $f_{te}$ that produces initial text embeddings $E_t$, followed by a graph encoder $f_{ge}$ that models hierarchical relations:
$$
\hat{E}_{t,v}^l = f_{ge}(E_{t,v}, l, \{E_{t,u}: u \in \mathcal{N}_v\}).
$$
The resulting hierarchical text embedding $\hat{E}_t$ is then used for classification by similarity with point embeddings [2508.09470]. The graph encoder is described as capturing semantic content from text, hierarchical granularity, parent-child relationships, and sibling similarity and difference.

This hierarchy-aware text branch is the mechanism that supports open-vocabulary behavior. The text encoder provides semantic class descriptions and a shared embedding space for labels, and new categories can be added by inserting a new leaf node into the graph, computing its hierarchical embedding, and applying the same similarity-based classifier [2508.09470]. The paper is explicit that this procedure enables category extension without redesigning the model.

## 4. Training objective and open-vocabulary inference

CitySeg uses a two-stage training strategy intended to stabilize optimization under multi-domain semantic inconsistency [2508.09470]. In Stage 1, all labels are merged into five base categories. For a point $\mathbf{p}_i$, point embedding $E_{p,i}$, and hierarchical text embedding $\hat{E}_{t,i}^l$, the classification probability is
$$
prob(y_i \mid \mathbf{p}_i) =
\frac{\exp(\sigma(E_{p,i},\hat{E}_{t,i}^l)/\tau)}
{\sum_{j=1}^{|y|}\exp(\sigma(E_{p,i},\hat{E}_{t,j})/\tau)},
$$
where $\sigma(\cdot,\cdot)$ is cosine similarity and $\tau$ is temperature. The loss in this stage is Cross-Entropy loss $L_{CE}$, with the stated purpose of learning broad urban semantics first and establishing a stable initialization for finer recognition [2508.09470].

In Stage 2, the model learns all subclasses and focuses on confusing sibling categories. The total loss is
$$
L = L_{CE} + \alpha L_h,
$$
where $\alpha$ balances a hinge loss term. For point embedding $E_{p,i}$, ground-truth hierarchical text embedding $\hat{E}_{t,i}^l$, and sibling set $\mathcal{S}$, the hinge loss is
$$
L_h = dist(E_{p,i},\hat{E}_{t,i}^l) + \sum_{j=1}^{|\mathcal{S}|}\max\left(m - dist(E_{p,i},\hat{E}_{t,j}^l), 0\right),
$$
with $dist(\cdot,\cdot)$ denoting Euclidean distance and $m$ a margin [2508.09470]. The intended effect is to pull a point embedding toward the correct label embedding while pushing it away from sibling categories by at least the margin.

The paper presents this hinge term as a remedy for fine-grained sibling collapse under ordinary cross-entropy. In ablation, adding hinge loss improves OA from $91.4$ to $93.6$ and mIoU from $63.5$ to $67.8$ [2508.09470]. The open-vocabulary setting follows directly from the same similarity rule used during training: zero-shot inference is performed by encoding point embeddings, comparing them to hierarchical text embeddings, and predicting the most similar class. The model therefore performs open-vocabulary segmentation without relying on visual information, a property the paper identifies as especially relevant for LiDAR-only UAV systems [2508.09470].

## 5. Benchmarks, implementation, and empirical performance

CitySeg is jointly trained on nine city-scale benchmarks: SensatUrban, Campus3D, STPLS3D, Swiss3Dcities, Hessigheim 3D, SUM, DALES, DublinCity, and ECLAIR, with UrbanBIS used as a separate open-set test domain [2508.09470]. The implementation uses Pointcept, a pre-trained PTv3 backbone, and a frozen CLIP text encoder. Training uses 8 NVIDIA A800 GPUs, batch size 16, 100 epochs, AdamW, learning rate $0.005$, OneCycleLR, and inputs consisting of 3D coordinates plus color. The local and global point counts per batch are 65,536 and 655,360 respectively, and the hinge-loss weight is $\alpha=0.3$ [2508.09470].

| Evaluation setting | Result |
|---|---|
| SensatUrban, PTv3 | 93.9 OA / 68.4 mIoU |
| SensatUrban, CitySeg | 95.2 OA / 72.0 mIoU |
| UrbanBIS, PTv3 + PPT | 53.7 OA / 42.6 mIoU |
| UrbanBIS, CitySeg-FM | 89.7 OA / 65.1 mIoU |
| UrbanBIS, fully supervised CitySeg | 92.0 OA / 70.4 mIoU |

The paper distinguishes two evaluation modes: CitySeg-FM, which uses shared weights across datasets, and CitySeg-FT, which is fine-tuned on each dataset [2508.09470]. Its overall claim is that CitySeg-FM already beats prior multi-dataset methods on most benchmarks, while CitySeg-FT achieves best results on all nine benchmarks. The most explicit per-dataset improvement reported in the data block is on SensatUrban, where PTv3 obtains $93.9$ OA / $68.4$ mIoU and CitySeg obtains $95.2$ OA / $72.0$ mIoU, corresponding to gains of $+1.3$ OA and $+3.6$ mIoU [2508.09470].

The UrbanBIS result is the strongest zero-shot demonstration. UrbanBIS is held out during training, and CitySeg-FM reaches $89.7$ OA / $65.1$ mIoU, compared with $53.7$ OA / $42.6$ mIoU for PTv3 + PPT [2508.09470]. The paper also notes that this zero-shot model remains only $2.3$ OA points below the fully supervised CitySeg upper bound on UrbanBIS, reported as $92.0$ OA / $70.4$ mIoU. This is the basis for the statement that, for the first time, CitySeg enables zero-shot generalization in city-scale point cloud scenarios without relying on visual information [2508.09470].

A plausible implication is that CitySeg’s empirical contribution lies less in a single architectural novelty than in the coordinated treatment of density normalization, cross-scale context, label hierarchy, and text alignment. The reported results are organized to support precisely that interpretation.

## 6. Terminology, adjacent usages, and conceptual context

The term “CitySeg” is not used uniformly across the literature, and distinguishing these usages is important. CitySeg in the strict sense refers to the 3D open-vocabulary semantic segmentation foundation model for city-scale point clouds described above [2508.09470]. By contrast, CitySeg/MOP is a benchmark test suite that turns hardware-aware neural architecture search for real-time semantic segmentation into fifteen standard multi-objective optimization problems derived from the Cityscapes dataset; it is a benchmark for optimization over segmentation architectures, not a 3D segmentation model [2404.16266].

Within semantic segmentation proper, CitySeg is also distinct from 2D urban-scene models such as SegRet and ProtoSeg. SegRet is an efficient encoder-decoder design for semantic segmentation that uses a Vision RetNet backbone and a lightweight residual decoder with zero-initialization, reporting Cityscapes results such as $81.75\%$ mIoU under single-scale evaluation for its Tiny model with $14.01$M parameters [2502.14014]. ProtoSeg, by contrast, is an interpretable semantic segmentation model based on class-specific prototypes and training-set patches, evaluated on Pascal VOC and Cityscapes, with Cityscapes mIoU of $67.23$ on validation and $67.04$ on test for the ImageNet-pretrained version [2301.12276]. These are image-based 2D approaches; CitySeg is a point-cloud foundation model for UAV-scale 3D scenes.

A further source of ambiguity is the broader “CitySeg-related” literature on urban segregation and city partitioning. MotifGPL studies urban social segregation through prototype-based graph structure extraction, motif distribution discovery, and urban graph reconstruction at the city-block level [2412.18464]. Other work measures socio-spatial segregation through multilayer transport-network random walks [2309.11901], mobility-derived experienced segregation across rings and pockets in U.S. cities [2407.12612], and imagery-based perception of exposure segregation through the VISAGE framework [2606.21858]. These are city-level segmentation or segregation analysis methods in the urban-science sense, not semantic segmentation models in computer vision.

This terminological divergence matters because CitySeg sits at the intersection of several neighboring traditions—city-scene parsing, foundation-model design, cross-dataset taxonomy alignment, and urban-scale spatial analysis—while belonging unambiguously to only one of them: city-scale 3D semantic segmentation for UAV perception [2508.09470].

Source: https://www.emergentmind.com/topics/cityseg