CM2LoD3: Automated LoD3 Reconstruction

• The paper introduces CM2LoD3, a method that learns facade semantics from conflict maps to reconstruct semantically enriched LoD3 building models.
• It employs a U-Net segmentation architecture fused with image-based methods to robustly detect openings like windows and doors under uncertainty.
• The technique outputs CityGML-consistent models that support urban planning, digital twins, and disaster management with improved facade-level detail.

CM2LoD3 is a method for automatically reconstructing LoD3 building models by learning facade semantics directly from Conflict Maps (CMs) derived via ray-to-model-prior analysis and, when available, fusing those semantics with image-based segmentation of textured 3D models under uncertainty (Hanke et al., 21 Aug 2025). Its target output is a CityGML-consistent LoD3 model in which LoD2 wall surfaces are enriched with semantic openings such as bldg:Window and bldg:Door, while preserving watertight topology. The method is motivated by the asymmetry between the broad availability of LoD1 and LoD2 building inventories and the relative scarcity of LoD3 models, whose facade-level detail is required for urban planning, digital twins, and disaster management.

1. Problem setting and conceptual scope

LoD1 and LoD2 building models are widely available, but they are not equivalent to LoD3. In the formulation used by CM2LoD3, LoD1 extrudes building footprints to volumes, whereas LoD2 adds roof geometry. These models are described as watertight, low-poly, georeferenced, and semantically organized through formats such as CityGML and CityJSON, yet they lack facade elements. That omission constrains applications requiring facade-level structure, including daylighting and thermal exchange analysis, visibility and line-of-sight reasoning for autonomous navigation, flood simulation, and access-point reasoning in disaster response (Hanke et al., 21 Aug 2025).

LoD3 addresses this gap by incorporating facade elements such as windows, doors, balconies, and underpasses. The central difficulty is that robust automation has historically depended on extensive, clean coverage with tightly registered multi-view images and dense point clouds. CM2LoD3 departs from that assumption by treating conflict-based evidence from terrestrial laser scanning as the primary signal for openings. In this formulation, the method does not begin with semantic point clouds or semantic images; instead, it makes Conflict Maps semantic.

A common misconception is that a Conflict Map already contains object semantics. In the CM2LoD3 formulation, this is explicitly not the case. Prior CM-based methods identify likely openings through contradictions between sensor rays and a building prior, but those maps are semantic-absent and previously required semantic point cloud or image segmentations with Bayesian post-processing. CM2LoD3 redefines the role of the CM by learning the classes {facade, window, door, unknown} directly on the CM raster itself (Hanke et al., 21 Aug 2025).

2. Conflict Maps and Semantic Conflict Map generation

A Conflict Map is a façade-aligned raster or texture in which each pixel encodes the visibility-based state of the LoD2 prior relative to terrestrial laser scanning rays. CM2LoD3 uses three pixel states. A confirmed pixel indicates that the LoD2 façade is supported by a TLS point within tolerance $t$. A conflict pixel indicates that the TLS ray penetrates the LoD2 surface and that the TLS point lies behind the façade surface, making the pixel a candidate opening. An unknown pixel indicates the absence of a consistent measurement because of occlusion, foreground clutter, or scan gaps (Hanke et al., 21 Aug 2025).

The CM is generated by raycasting from scanner viewpoints to TLS points and comparing intersection distances to LoD2 surfaces. The method marks each prior surface pixel as confirmed, conflict, or unknown under a tolerance empirically set to $t = \pm 0.7\,\text{m}$ to accommodate LoD2 façade misalignments and minor bulges. The paper states that this tolerance must be high because of the LoD2 model and that the aim is to detect windows and doors as building openings (Hanke et al., 21 Aug 2025).

CM2LoD3 extends the CM into a Semantic Conflict Map through supervised segmentation. Because large quantities of real labeled CMs are not readily available, the method introduces the Semantic Conflict Map Generator (SCMG). SCMG uses Random3DCity to produce varied façade layouts, renders synthetic CMs, and augments them with controlled occlusions so that synthetic training data better matches real TLS phenomena such as vegetation and street clutter. The generator includes 228 tree silhouettes and 426 random masks, and it maps CMP façade labels to the four classes {facade, window, door, unknown}. The ambiguous class shop is merged into background where appropriate to reduce ambiguity (Hanke et al., 21 Aug 2025).

The occlusion model is not an incidental augmentation; it is part of the generalization strategy. Tree masks are positioned with a Gaussian model and a grounding offset:

$x_p = \mu_x + \sigma_x \cdot \mathcal{N}(0,1)$

with

$$\mu_x = \frac{W_{cm} - W_{tree}}{2}, \qquad \sigma_x = \frac{W_{cm} - W_{tree}}{6},$$

subject to

$0 \leq x_p \leq W_{cm} - W_{tree},$

and

$y_p = H_{cm} + \delta - H_{tree}.$

This design is described as creating natural clustering and grounded placement (Hanke et al., 21 Aug 2025). A plausible implication is that SCMG is intended not merely as synthetic data generation, but as a domain-bridging mechanism between idealized procedural facades and the structured incompleteness of real TLS-derived CMs.

3. Segmentation architecture and uncertainty-aware fusion

The CM segmentation backbone is an adapted U-Net operating on 3-channel CM inputs. The implementation preserves pixel-wise spatial fidelity through same-padding, size-matching interpolation, explicit dimension checks for skip connections, and encoder-decoder symmetry with skip connections. The classifier predicts one semantic label per pixel in the set {facade, window, door, unknown} (Hanke et al., 21 Aug 2025).

Training combines synthetic SCMG data with a small portion of real TLS CMs for domain robustness. The paper reports a training composition with approximately 10% real TLS CMs and gives the training schedule explicitly: Adam with learning rate $2 \times 10^{-4}$, weight decay $1 \times 10^{-5}$, StepLR with step=10 and $\gamma=0.5$, batch size 14, and early stopping with patience=8 and $\delta = 5 \times 10^{-5}$. Convergence is reported at 41 epochs with training loss 0.0098 and validation loss 0.0116 (Hanke et al., 21 Aug 2025).

The optional second modality is image-based facade segmentation on textured models. The paper applies Mask R-CNN, SegFormer, and Mask2Former to facade images, producing per-pixel probabilities, especially for windows and doors. CM2LoD3 does not treat this branch as a replacement for CM semantics; it is a complementary modality whose outputs are fused with CM probabilities by class-specific confidence weighting (Hanke et al., 21 Aug 2025).

The fusion equations are stated as

$t = \pm 0.7\,\text{m}$0

$t = \pm 0.7\,\text{m}$1

with $t = \pm 0.7\,\text{m}$2 per class and fixed coefficients chosen empirically from validation (Hanke et al., 21 Aug 2025). The stated rationale is that the stronger modality can be emphasized separately for each class; the paper gives the example that Mask R-CNN can be stronger for windows in textured facades. This class-specific fusion is designed to improve robustness under transparent glass, occlusion, reflections, and multi-layered facade structures.

The effect of mixed training is quantified for doors: including real TLS CMs improves door IoU from 0.059 with GEN + CMP only to 0.142 with GEN + CMP + REAL (Hanke et al., 21 Aug 2025). This is a direct indication that domain shift between synthetic and real CMs is material and that even a small amount of real data can substantially alter class separability, especially between door, unknown, and facade.

4. Semantic 3D reconstruction and CityGML insertion

The reconstruction stage starts from LoD2 wall surfaces and turns 2D semantic predictions into 3D openings. After segmentation and optional fusion, CM2LoD3 post-processes masks with morphological opening using kernel (5,5) to remove small blobs; shape approximation is optional. For each wall surface, 2D mask coordinates are normalized to wall extents and mapped into 3D with bilinear interpolation from the wall corners $t = \pm 0.7\,\text{m}$3 (Hanke et al., 21 Aug 2025):

$t = \pm 0.7\,\text{m}$4

This mapping provides the 3D geometry for opening polygons. CM2LoD3 then inserts those polygons as interior rings in the wall B-Rep. The winding order for interior rings is reversed, and a semantic element bldg:Window or bldg:Door is created with geometry matching the opening polygon. The procedure enforces closed loops and the absence of self-intersections so that the final solid remains watertight and CityGML-consistent (Hanke et al., 21 Aug 2025).

The reconstruction logic is notable because it preserves both geometry and semantics. The output is not simply a mesh with holes; it is a semantically structured building model in which openings are attached in the CityGML hierarchy to the corresponding wall surfaces. This suggests that CM2LoD3 is aimed at downstream urban-information workflows rather than only visual reconstruction.

At the workflow level, the paper describes the pipeline as follows: visibility analysis and CM creation; SCMG synthetic data generation; CM segmentation model training; optional texture segmentation; uncertainty-aware fusion; and semantic 3D reconstruction. The required inputs are a LoD2 building prior, TLS point clouds and viewpoints, optional facade images, and synthetic SCMs from SCMG; the outputs are semantic CMs, fused segmentation with confidence, and LoD3 building models with semantically inserted openings (Hanke et al., 21 Aug 2025).

5. Evaluation protocol, quantitative results, and empirical behavior

Evaluation uses both real and synthetic data. The real dataset consists of TLS and facade images from TUM2TWIN, covering 4 buildings around the TUM campus with 272 annotated openings and 3 facade images for fusion examples. Synthetic data are generated through SCMG using Random3DCity layouts, the occlusion augmentations described above, and the remapped CMP dataset (Hanke et al., 21 Aug 2025). The primary metric is Intersection over Union with per-class breakdown, supplemented by qualitative inspection because the ground truth does not fully represent all facade detail, such as window sashes.

The main reported quantitative results emphasize window segmentation and the effect of fusion:

Method	Window IoU
U-Net (SCM only)	0.178
Mask R-CNN	0.357
Fusion	0.467
Fusion (with shape approximation)	0.420

These values show that the fused prediction outperforms either individual modality on the reported window IoU benchmark (Hanke et al., 21 Aug 2025). The abstract also reports “the 61% performance with uncertainty-aware fusion of segmented building textures,” and the paper describes this as highlighting the benefit of fusion.

The ablation findings are methodologically specific. Adding real TLS CMs reduces misclassification between unknown and facade. Removing the CMP shop class improves window detection stability. Artificial occlusions during training improve robustness to real occlusions (Hanke et al., 21 Aug 2025). Together, these ablations indicate that the system’s performance depends not only on network architecture but also on how semantics are defined and on whether synthetic generation reproduces occlusion statistics seen in TLS.

The failure cases are equally instructive. Real TLS CMs can be noisy and may produce non-rectangular window shapes. Mask R-CNN can blur window predictions or misclassify doors and windows. Reflective glass, heavy occlusions, and facade clutter reduce confidence for both modalities. Fusion mitigates but does not eliminate these issues (Hanke et al., 21 Aug 2025). Another evaluation caveat is that differences in ground-truth granularity can depress IoU for CM-only U-Net predictions even when they contain qualitatively richer structural detail than the annotations. This is especially relevant for sash-level structure that is not represented in the ground truth.

6. Significance, limitations, and prospective extensions

CM2LoD3 positions scalable LoD3 reconstruction as an enrichment problem over existing LoD2 inventories. Its applications are explicitly framed around urban planning, digital twins, and disaster management. In urban planning, the enriched models support daylight and ventilation analysis, visibility studies, and energy simulations. In digital twins, they support autonomous-driving occlusion reasoning, facade maintenance, and urban analytics. In disaster management, they support access-point detection, evacuation modeling, and flood ingress simulation (Hanke et al., 21 Aug 2025). Because the method begins from broadly available LoD2 data and requires only CMs plus optional textures, it is presented as a practical path toward city-scale LoD3 reconstruction.

The limitations are equally explicit. CM accuracy depends on TLS quality, viewpoint correctness, and registration. The training distribution is biased toward European architecture, so generalization to other styles may require broader SCMG sampling. The present fusion stage uses fixed coefficients rather than adaptive or learned reliabilities. The paper also identifies extensions toward additional facade elements such as balconies, blinds, and air conditioners; multi-view integration; stronger rectangularity priors; stronger uncertainty modeling; and the use of foundation models and larger datasets for semantic consistency (Hanke et al., 21 Aug 2025).

A further misconception is that CM2LoD3 is exclusively an image-segmentation pipeline with a geometric postprocessor. The method is better understood as a conflict-centric semantic reconstruction pipeline in which image segmentation is optional and secondary. Its defining move is to convert ray-to-prior contradictions into a learnable semantic signal and then to insert the resulting openings into a CityGML-consistent LoD3 representation. In that sense, the principal contribution is not only better 2D segmentation, but a reformulation of how facade openings can be inferred from the interaction between sensing rays and a geometric prior (Hanke et al., 21 Aug 2025).