RoofSeg: Roof Segmentation & Geometry Analysis

• RoofSeg is a roof-focused segmentation framework that recovers detailed roof facets and geometric attributes from remote sensing data.
• It employs edge-aware transformers for LiDAR-based end-to-end roof plane segmentation and Mask R-CNN for joint mask and geometric regression in orthophotos.
• Evaluations across diverse datasets show high accuracy and robust geometric fidelity, supporting advanced 3D building reconstruction pipelines.

Searching arXiv for papers directly about “RoofSeg” and closely related roof segmentation/reconstruction work. RoofSeg denotes roof-focused segmentation systems that recover roof facets, masks, planes, or roof-centered geometric attributes from remote sensing data. In arXiv usage, the name is explicit for an edge-aware transformer-based network for end-to-end roof plane instance segmentation from airborne LiDAR point clouds (You et al., 26 Aug 2025), and it is also a natural label for a Mask R-CNN-based system that jointly predicts roof-segment masks together with building height, roof slope, and roof azimuth from a single aerial orthophoto (Versteeg et al., 25 May 2026). More broadly, the term sits within a methodological family that includes roof-plane clustering in point clouds, edge-based polygonization from aerial imagery, single-image roof-section reconstruction, and roof-specialized semantic or instance segmentation under strong geometric and class-imbalance constraints (Li et al., 2023, Mei et al., 12 Mar 2025, Lussange et al., 2023, Boin et al., 2020).

1. Scope and nomenclature

The literature uses “RoofSeg” in more than one sense. The most literal usage is the 2025 method titled “RoofSeg: An edge-aware transformer-based network for end-to-end roof plane segmentation,” which addresses per-point roof plane instance segmentation from airborne LiDAR point clouds (You et al., 26 Aug 2025). A second, closely related usage is a roof-segment instance segmentation and geometry regression system from a single aerial orthophoto, where the network predicts a mask plus three continuous geometric attributes—building height, roof slope, and roof azimuth—for each roof facet (Versteeg et al., 25 May 2026). Adjacent papers use the term more generically to denote roof-centric segmentation pipelines rather than a single canonical architecture.

Use of “RoofSeg”	Input modality	Principal output
RoofSeg as a named LiDAR method	Airborne LiDAR point clouds	End-to-end roof plane instance labels (You et al., 26 Aug 2025)
RoofSeg as joint mask-and-geometry inference	Single aerial orthophoto	Roof-segment masks, height, slope, azimuth (Versteeg et al., 25 May 2026)
RoofSeg as a broader roof-analysis pipeline	Aerial or satellite imagery, point clouds	Roof polygons, roof sections, or roof-focused semantic/instance masks (Mei et al., 12 Mar 2025, Lussange et al., 2023)

This multiplicity matters because the common thread is not a single network family, but a recurring research objective: to elevate roof extraction from coarse building detection to roof-facet-level geometric understanding. That objective appears in image-space instance segmentation, point-cloud plane decomposition, vector polygonization, and downstream LoD2 reconstruction.

2. RoofSeg as an end-to-end LiDAR roof-plane segmentor

The named RoofSeg model operates on roof-level airborne LiDAR point clouds and targets truly end-to-end roof plane instance segmentation, rather than feature learning followed by external clustering (You et al., 26 Aug 2025). Each roof is resampled to 2048 points. The backbone is PointNet++ with four Set Abstraction layers and four Attention-based Feature Propagation layers, all attention blocks using 256-dimensional features. Plane queries are generated by Farthest Point Sampling and Fourier positional encoding, then refined by eight Query Refinement Decoder blocks. Instance masks are produced by query–point affinity, while a query semantic branch scores which queries correspond to valid roof planes.

Two components distinguish the method. First, the Edge-Aware Mask Module predicts edge points, fits a plane to interior points via PCA, computes point-to-plane distances for edge points, and fuses these geometric priors with point features before mask refinement. Second, the loss couples an adaptive weighting strategy in the mask loss with a plane geometric loss that penalizes average point-to-plane distance, alongside semantic classification loss and an edge-mask loss. Hungarian matching assigns predicted masks to ground-truth plane instances during training. The formulation is explicitly designed to address three stated weaknesses of earlier deep roof-plane methods: lack of true end-to-end optimization, weak discriminability near plane boundaries, and insufficient geometric constraint during training (You et al., 26 Aug 2025).

The method is evaluated on RoofNTNU, Roofpc3D, and a roof-plane subset of Building3D. The reported metrics are mCov, mWCov, mPrec, and mRec.

Dataset	mCov / mWCov	mPrec / mRec
RoofNTNU	0.9589 / 0.9682	0.9960 / 0.9818
Roofpc3D	0.9601 / 0.9756	0.9998 / 0.9854
Building3D	0.9374 / 0.9637	0.9879 / 0.9658

Ablations show that EAMM, the adaptive mask loss, and the plane geometric loss each contribute measurable gains, with the full configuration outperforming variants lacking edge-aware refinement or geometric supervision. Additional ablations indicate that PointNet++ with attention-based feature propagation outperforms a 3D U-Net backbone and a PointNet++ backbone with standard feature propagation, and that eight decoders form the preferred accuracy–efficiency trade-off (You et al., 26 Aug 2025). In this sense, RoofSeg is not merely a Transformer insertion into a standard point pipeline; it is a query-based instance segmentation framework whose training objective is explicitly geometric.

3. RoofSeg as joint roof-segment masking and geometric attribute regression

A second major instantiation of RoofSeg is the orthophoto-based system of “Joint Instance Segmentation and Geometric Attribute Regression for Roof Structures in Aerial Imagery” (Versteeg et al., 25 May 2026). Its task is to learn

$$f_\theta: I \rightarrow \{(b_i, m_i, h_i, \alpha_i, \phi_i)\}_{i=1}^{N},$$

where each roof segment instance has a bounding box, binary mask, building height $h$, roof slope $\alpha$, and roof azimuth $\phi$. The method works at the level of roof facets rather than whole buildings, following the LoD2.2 decomposition in 3DBAG, where a building can contain multiple planar roof surfaces corresponding to different ridges, dormers, or slopes.

The dataset is constructed from PDOK Luchtfoto RGB orthophotos and 3DBAG LoD2.2 roof segments. Each sample is a 100 m × 100 m crop centered on a building centroid, resized to $1024 \times 1024$ pixels, with 3DBAG polygons converted to instance masks and per-instance attributes. The full dataset contains 17,353 images and 2,110,801 roof segment instances, with geographically separated train, validation, and test splits of 11,931, 2,747, and 2,675 images respectively (Versteeg et al., 25 May 2026).

Architecturally, the system extends Mask R-CNN with FPN by adding a geometric attribute regression branch. The best backbone is DINOv3 ConvNeXt-Base, which outperforms ResNet-50 and EfficientNetV2-M, especially at $\mathrm{AP}_{75}$. The attribute branch uses the same $7 \times 7$ RoI features as the classification and box heads, followed by four convolutional layers with BatchNorm and a two-layer MLP with 128 hidden units and ReLU. It outputs height, roof angle, and azimuth in a biternion form $(\cos\phi, \sin\phi)$, decoded by $\operatorname{atan2}$. Hungarian matching based on mask IoU assigns predictions to ground truth, and unmatched predictions are ignored for attribute regression (Versteeg et al., 25 May 2026).

Two design choices are central. The first is a conditional azimuth loss that supervises azimuth only when the ground-truth slope exceeds $15^\circ$, because flat roofs in 3DBAG have arbitrary azimuth values. In ablation, this reduces azimuth MAE from approximately $h$0 to approximately $h$1. The second is a log-normalized height target

$h$2

with $h$3 and $h$4, introduced to handle the strongly right-skewed distribution of building heights. In ablation, height MAE drops from 1.878 m for raw heights to 1.556 m for log-normalized heights (Versteeg et al., 25 May 2026).

Training uses a two-stage schedule: eight epochs of Mask R-CNN detection and mask training only, followed by eight epochs with the backbone frozen while training the bounding-box, classification, mask, and attribute heads jointly. Adam is used with learning rates $h$5 for the backbone and $h$6 for the other heads, batch size 16 on 4×A100 GPUs, mixed precision, and dihedral plus color-jitter augmentation (Versteeg et al., 25 May 2026).

At $h$7 input resolution, the best configuration reaches $h$8, $h$9, and mAP $\alpha$0, with height MAE approximately 1.03 m, slope MAE approximately $\alpha$1, and azimuth MAE approximately $\alpha$2 (Versteeg et al., 25 May 2026). The predicted masks and attributes are sufficient to reconstruct simplified LoD2 3D building models from a single overhead image, with LiDAR-derived 3DBAG data required only for training.

4. Related methodological lineages

Several adjacent papers clarify how RoofSeg fits into a broader technical landscape. In point clouds, “A boundary-aware point clustering approach in Euclidean and embedding spaces for roof plane segmentation” learns semantic labels, point offsets, and 64-dimensional embeddings with a three-branch PointNet++ network, then clusters interior plane points jointly in shifted Euclidean space and embedding space before assigning boundary points by a combined point-to-plane and embedding distance (Li et al., 2023). That method is not end-to-end in the same sense as RoofSeg (You et al., 26 Aug 2025), but it establishes the importance of boundary-aware processing and dual-space clustering for roof facets.

In aerial imagery, “Polygonizing Roof Segments from High-Resolution Aerial Images Using Yolov8-Based Edge Detection” takes an explicitly edge-centric route: YOLOv8-OBB predicts roof edges, DBSCAN clusters endpoints into junctions and gaps, rasterized edges are grouped by connected components, and per-face polygons are assembled for vector output (Mei et al., 12 Mar 2025). Its evaluations combine raster metrics such as mIoU and ovIoU with vector metrics such as Hausdorff distance, PolyS, and the composite raster–vector metric $\alpha$3. This line of work treats roof segmentation less as dense pixel labeling and more as topologically consistent polygon recovery.

Single-image and multi-view 3D roof reconstruction papers provide additional context. KIBS performs Mask R-CNN-based 2D segmentation of roof sections, blends those masks into the RGB image, then uses a second Mask R-CNN to infer discrete corner-height classes from a single non-orthogonal satellite image, yielding LOD2 reconstruction with Jaccard indices of 88.55% and 75.21% on its two datasets and height mean errors of 1.60 m and 2.06 m (Lussange et al., 2023). “3D Instance Segmentation of MVS Buildings” instead starts from multi-view RGBH images, segments roof instances in 2D, clusters masks across views with an occlusion-aware similarity, and back-projects them to 3D roof instances before extending them to full buildings through an MRF (Chen et al., 2021). Taken together, these papers show that RoofSeg can appear either as a direct predictor of roof planes or as the first stage of a larger 3D reconstruction system.

5. Datasets, task variants, and application domains

RoofSeg research spans a wide variety of datasets and task formulations. ZRG is a large-scale residential rooftop dataset with 22,334 properties, 425,660 roof faces, high-resolution orthomosaics, DSMs, colored point clouds, and 3D roof wireframe annotations; its baselines include roof outline extraction, monocular height estimation, and planar roof structure extraction, with DeepLabV3+ reaching mIoU 96.83 for roof outline extraction and Mask R-CNN reaching mAP 72.1 for face instance segmentation on ZRG-Test (Corley et al., 2023). Nacala-Roof-Material contributes 17,954 buildings from Mozambique, five roof material classes, and a multi-task benchmark for object detection, classification, and segmentation; it also introduces a DOW variant that improves the trade-off between semantic segmentation and instance segmentation for both U-Net and DINOv2 backbones (Guthula et al., 2024).

Task specialization is equally varied. Roof damage segmentation on xBD emphasizes severe class imbalance and shows that class-aware oversampling alone raises macro F1 from 17.4% to 53.3%, while the full model reaches 57.3% (Boin et al., 2020). Fine-grained building roof instance segmentation in the 2023 IEEE GRSS Data Fusion Contest uses a Cascade Mask R-CNN with a composite dual-backbone and domain adapted pretraining, reaching test $\alpha$4 (Liu et al., 2023). Roofpedia focuses on sustainable roof typology, building an open registry for more than one million buildings across 17 cities by combining U-Net-based solar and green roof segmentation with geospatial postprocessing (Wu et al., 2020).

Applications are correspondingly broad. Roof segmentation and roof-object segmentation can be coupled with structured-data pitch prediction, azimuth estimation, shading masks, and pvlib/PVWatts modeling to estimate rooftop solar potential at large scale (Soares et al., 2021). Roof shape classification from fused satellite and LiDAR features reaches 98.6% test accuracy with a Random Forest on dual fused features, motivated by multicopter emergency landing site selection (Castagno et al., 2018). For historical aerial imagery, a GAN-enhanced pipeline using DeOldify and Real-ESRGAN raises rooftop detection performance, with YOLOv11n achieving mAP exceeding 85% and improving by approximately 40% over original black-and-white images (Chen et al., 29 Mar 2025). A plausible implication is that RoofSeg is best understood not as a single benchmark problem but as a family of roof-centric perception tasks sharing a common need for geometric faithfulness at roof-facet scale.

6. Limitations and likely research directions

The principal limitations are strongly modality dependent. In the orthophoto-based system, performance may degrade outside the Dutch domain because training data is entirely Dutch; errors are larger for very tall buildings, inward-sloping roofs, unusual geometries, temporally misaligned imagery and 3DBAG, and over-segmented ground truth that does not match visually coherent roof surfaces (Versteeg et al., 25 May 2026). In the LiDAR RoofSeg model, efficiency remains a concern because eight Transformer decoders and multiple attention modules are computationally nontrivial, while sparse or noisy point clouds and non-planar roof elements can still degrade performance (You et al., 26 Aug 2025). Edge-based polygonization remains vulnerable to full occlusion and to texture-induced false edges, particularly when material changes mimic structural boundaries (Mei et al., 12 Mar 2025).

Several broader findings recur across the literature. Point cloud density, density uniformity, and 3D point precision materially influence roof plane segmentation quality (Zeng et al., 30 May 2025). Multimodal fusion is not automatically beneficial: in fine-grained roof instance segmentation, naive RGB+SAR fusion under a dual-backbone architecture degraded performance relative to RGB-only training (Liu et al., 2023). Off-nadir polygon extraction remains sensitive to prompt quality, downsampling, and small-building resolution limits, even when high-quality mask prompting and offset modeling are used (Li et al., 2024).

This suggests three converging directions. First, more geometry-aware learning objectives will likely continue to replace purely appearance-driven losses, whether through plane-distance penalties, boundary-aware masking, or conditional supervision. Second, tokenization quality—superpoints in point clouds, edge graphs in vectorization, or instance masks in orthophotos—appears to be as important as the downstream network class. Third, future RoofSeg systems will likely couple segmentation more tightly with downstream structure recovery, so that masks, planes, polygons, and LoD2 building models are optimized as parts of a single geometric pipeline rather than as loosely connected postprocessing stages.