OpenUrban3D: Urban 3D Segmentation

• OpenUrban3D is an annotation-free open-vocabulary semantic segmentation framework for large-scale urban 3D point clouds, enabling zero-shot segmentation via arbitrary text queries.
• It employs multi-view, multi-granularity rendering combined with mask-level vision-language feature extraction and sample-balanced fusion to derive robust semantic features.
• A 2D-to-3D distillation process integrates CLIP-aligned semantic signals into a 3D backbone, leading to significant improvements on urban segmentation benchmarks.

OpenUrban3D is an annotation-free, open-vocabulary semantic segmentation framework for large-scale urban 3D point clouds. Open-vocabulary semantic segmentation enables models to recognize and segment objects from arbitrary natural language descriptions, offering the flexibility to handle novel, fine-grained, or functionally defined categories beyond fixed label sets. In the urban domain, where high-quality, well-aligned multi-view imagery is frequently absent and existing three-dimensional segmentation pipelines generalize poorly across diverse urban environments with substantial variation in geometry, scale, and appearance, OpenUrban3D generates robust semantic features directly from raw point clouds through multi-view, multi-granularity rendering, mask-level vision-language feature extraction, sample-balanced fusion, and distillation into a 3D backbone model, enabling zero-shot segmentation for arbitrary text queries while capturing both semantic richness and geometric priors (Wang et al., 13 Sep 2025).

1. Scope, motivation, and problem formulation

Large-scale urban point clouds support applications such as digital twins, smart city management, and urban analytics. The main obstacles for open-vocabulary semantic segmentation in this setting are the frequent absence of high-quality, well-aligned multi-view imagery in large-scale urban point cloud datasets and the poor generalization of existing 3D segmentation pipelines across diverse urban environments with substantial variation in geometry, scale, and appearance. OpenUrban3D addresses these constraints by operating without aligned multi-view images, pre-trained point cloud segmentation networks, or manual annotations (Wang et al., 13 Sep 2025).

The broader urban-systems context clarifies why such a framework matters. Urban digital twins are increasingly perceived as a way to pool the growing digital resources of cities, while models and simulations are central to this undertaking; open infrastructures such as the Urban Model Platform emphasize open standards, decentralized integration of models, and a multi-model approach to representing urban systems (Herzog et al., 12 Jun 2025). A survey of visual analytics for 3D urban data likewise identifies a shortage of scalable, open-source frameworks for large-volume, semantically rich 3D data (Miranda et al., 2024). This suggests that OpenUrban3D addresses a semantic bottleneck that is downstream of data acquisition and upstream of urban modeling, simulation, and visual analysis.

A recurring misconception in urban 3D semantics is that open-vocabulary behavior in practice still depends on manually curated supervision or well-aligned image collections. OpenUrban3D departs from that premise by using only raw point cloud data and by constructing its semantic signal through synthetic views, vision-language alignment, and 2D-to-3D distillation rather than through manual labeling (Wang et al., 13 Sep 2025).

2. Pipeline architecture and multi-view rendering

The framework is organized as a modular pipeline that begins with raw geometry and ends with text-driven pointwise predictions.

Stage	Mechanism	Output
1	Multi-view, multi-granularity virtual image rendering	Rendered RGB views
2	2D mask-level vision-language feature extraction	Masks and CLIP-aligned features
3	Back-projection and sample-balanced fusion	Per-point 2D semantic features
4	2D-to-3D feature distillation	3D backbone features
5	Hybrid 2D-3D feature fusion for inference	Zero-shot segmentation

Given a point cloud $P \in \mathbb{R}^{N \times 3}$, OpenUrban3D renders multiple RGB images from diverse viewpoints and at multiple scales. For global views, virtual cameras are placed on a circle high above the scene centroid:

$$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$

with target point

$$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$

For local views, a grid with $(K+1)\times(K+1)$ anchor points covers the XY-plane, and each anchor spawns local cameras in a similar circular orbit targeting the local region. This systematic sampling ensures coverage of both large structures such as buildings and small structures such as cars. The number of views and density, angular interval $A$, radius $R$, and granularity $K$ are hyperparameters (Wang et al., 13 Sep 2025).

Each rendered image is then processed by a pre-trained mask-based 2D vision-LLM, such as ODISE or SAM+CLIP. The model outputs binary masks $\{M_k\}$ and associated CLIP-aligned feature vectors $\{\mathbf{f}_k\}$, encoding object-level semantics. At this stage, OpenUrban3D converts synthetic views into language-aligned semantic fragments, rather than attempting to learn open-vocabulary semantics directly from raw urban point sets (Wang et al., 13 Sep 2025).

3. Back-projection, sample-balanced fusion, and 2D-to-3D distillation

The 2D semantic signal is transferred back to 3D by projecting each point $\mathbf{p}$ into every rendered view using standard pinhole camera geometry. Occlusions are resolved by Z-buffer or depth map comparison. If a point projects into a valid mask in a view and passes the depth check, the corresponding feature is assigned to that point (Wang et al., 13 Sep 2025).

A central issue in urban scenes is object-size and frequency imbalance: buildings and ground dominate the sample distribution, while smaller or rarer classes are easily suppressed by naive fusion. OpenUrban3D therefore uses a sample-balanced multi-view feature fusion procedure. For each mask in a view, the framework counts its point assignments, computes a threshold $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$0 as the mean of the point counts of the $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$1 smallest masks, randomly down-samples masks with $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$2 to $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$3, and retains all associations for smaller masks. For points observed in multiple views after sampling, the 2D feature is averaged:

$$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$4

This balances the dataset at the feature fusion level and maintains semantic richness for small and rare classes (Wang et al., 13 Sep 2025).

The fused 2D semantic features then supervise a 3D backbone network, implemented with MinkUNet, through knowledge distillation:

$$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$5

Gradients are stopped for $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$6 so that only the 3D student is updated. In the formulation given for the framework, the fused 2D semantic features act as a “teacher” and the 3D point cloud encoder acts as a “student.” This distills CLIP-aligned semantic knowledge into the 3D geometry, even for points not visible in 2D. For implementation, the framework uses MinkUNet as the 3D backbone and the ODISE Image Encoder plus CLIP Text Encoder for vision-language alignment (Wang et al., 13 Sep 2025).

4. Zero-shot inference and open-vocabulary behavior

At inference time, the framework supports arbitrary text queries. The query may be a category name, a fine-grained phrase, or a functional description; an LLM can also parse complex instructions, mapping them to atomic category prompts. The CLIP Text Encoder generates text embeddings $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$7, and each point receives a fused semantic representation:

$$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$8

with a small $$\mathbf{C}^k = (c_x + r_g \cos\theta_k,\; c_y + r_g \sin\theta_k,\; H + \sqrt{L \cdot W}),$$9—typically $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$0—injecting geometric structure into strongly semantic 2D features. Pointwise prediction is then computed as

$$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$1

This yields annotation-free, open-vocabulary, zero-shot semantic segmentation (Wang et al., 13 Sep 2025).

The design explicitly separates two priors. The 2D mask features are used for semantic alignment and fine object recognition, whereas the 3D distilled features encode geometric regularities and augment points missing from synthetic views. Their hybrid fusion captures both visual semantics and spatial context or geometric consistency. In the reported characterization, 2D features dominate semantic understanding, but 3D features sharpen geometry, especially boundaries (Wang et al., 13 Sep 2025).

Open-vocabulary capability does not imply unconstrained semantic precision. The framework’s ablations show that prompt quality matters: accurate, specific prompts, possibly generated by LLMs, directly improve results. This is particularly relevant for high intra-class-variance categories and categories that are rare or small in urban scenes (Wang et al., 13 Sep 2025).

5. Benchmarks, metrics, and empirical performance

OpenUrban3D is evaluated on large-scale urban benchmarks with withheld ground truth labels. The reported datasets are SensatUrban and SUM.

Dataset	Dataset description	Reported OpenUrban3D results
SensatUrban	UAV photogrammetry, $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$2B points, $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$3 km$$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$4, 3 UK cities	mIoU 39.6%, OA 84.7%
SUM	Helsinki city, from oblique aerial images, 30 points/m$$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$5 used	mIoU 75.4%, OA 90.5%

The evaluation uses mIoU, mAcc, and OA, with

$$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$6

On SensatUrban, OpenUrban3D achieves mIoU 39.6% and OA 84.7%, a large improvement over prior 3D open-vocabulary methods such as OpenScene at 12.6% mIoU and PLA or RegionPLC at below 3%. The reported fully supervised MinkUNet baseline reaches 46.2% mIoU, leaving a narrow gap of 6.6%. On SUM, OpenUrban3D reaches mIoU 75.4% and OA 90.5%, exceeds all open-vocabulary baselines including OpenScene at 41.1%, and even outperforms state-of-the-art supervised models such as PTV3 at 74.0% (Wang et al., 13 Sep 2025).

Per-class behavior is uneven in a way that is typical of urban semantics. The framework is reported as excellent on well-defined categories such as buildings, vegetation, water, and boats. It is harder for high intra-class-variance classes such as “wall,” including fences and barriers, where prompt specificity matters, and for rare or small classes such as “bike” and “rail.” The ablation studies attribute part of the gain to view design and feature balancing: optimizing view density and granularity, with lower $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$7 and moderate $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$8, maximizes coverage and mIoU; sample-balanced fusion improves learning for small or rare classes, with “Footpath” mIoU rising from 0 to 1.4%; and hybrid feature-level fusion outperforms result-level ensemble, with a small 3D weight $$T_g = \left(c_x,\; c_y,\; \tfrac{1}{2}(H + \sqrt{L \cdot W})\right).$$9 optimal. Training is label-free, while evaluation uses withheld ground-truth labels (Wang et al., 13 Sep 2025).

6. Role in urban 3D semantics and adjacent research directions

OpenUrban3D has several immediate practical implications. It removes manual annotation from large-scale urban semantic mapping, supports arbitrary and previously unseen categories through open-vocabulary queries, works when colorization or imagery are sparse or missing, and demonstrates robustness to new cities, sensor types, and urban heterogeneity. Its LLM integration supports natural-language-driven, function-oriented urban planning or analytics and interactive scene understanding (Wang et al., 13 Sep 2025).

These properties are significant because semantically enriched 3D urban data is a recurring requirement across adjacent research areas. The visual-analytics literature emphasizes thematic layers linked to physical urban entities, with use cases spanning wind, temperature, noise, view impact, walkability, and digital twins (Miranda et al., 2024). The Urban Model Platform frames open, federated urban modeling as a socio-technical backbone for modeling and simulation in urban digital twins, built on open standards and decentralized model integration (Herzog et al., 12 Jun 2025). A plausible implication is that OpenUrban3D can serve as an annotation-free semantic layer within such infrastructures, where segmented urban objects become inputs to model registries, scenario tools, or comparative analyses.

OpenUrban3D is also complementary to more formal semantic urban data models. CityGML-oriented conversion workflows were developed to map semantic information onto geometric models for thematic queries, spatial data-mining, and geospatial simulation and analysis (Swarup, 2021). By contrast, OpenUrban3D starts from raw point clouds and infers semantics through open-vocabulary vision-language alignment. This suggests complementarity rather than substitution: CityGML-style pipelines organize semantically mapped geometry into explicit urban object schemas, whereas OpenUrban3D supplies a scalable route to obtaining semantic labels in the first place.

The framework’s limitations remain structurally informative. Current weaknesses are concentrated in classes with high intra-class variance, in rare and small categories, and in settings where prompt specificity is poor. Performance is also sensitive to view density, granularity, and fusion design. Those constraints indicate that open-vocabulary urban segmentation is not merely a language-alignment problem; it is also a problem of spatial coverage, class balance, and 2D-to-3D consistency. Within that framing, OpenUrban3D establishes a technically specific paradigm for annotation-free, zero-shot urban scene understanding at scale (Wang et al., 13 Sep 2025).