Edge Prediction for Roof Wireframe Reconstruction with Transformers

Abstract: This paper presents a competitive solution to the S23DR Challenge 2026, which aims to reconstruct 3D house roof wireframe models from sparse SfM point clouds and ground-level semantic segmentations and depth maps. Our proposed method utilizes an end-to-end Transformer encoder-decoder architecture inspired by DETR. To effectively process the geometric and semantic data, the sparse SfM point cloud input is dynamically subsampled based on semantic priority and augmented with Gestalt and ADE20k class features. To further increase segmentation context, we fuse the point features with additional Gestalt feature encodings which are obtained by projecting the points into latent feature maps produced by a frozen autoencoder. Learned query embeddings are then decoded directly into 3D wireframe edges via cross-attention mechanisms. Evaluated on the "HoHo 22k" dataset, our approach significantly outperforms both handcrafted and learned baselines, achieving a Hybrid Structure Score (HSS) of 0.6476 and securing the second-highest position on the challenge's private leaderboard.

• The paper introduces a transformer architecture that fuses sparse geometric and semantic cues for precise 3D roof wireframe reconstruction.
• It leverages multi-view Gestalt feature fusion and a DETR-style encoder-decoder to directly predict 3D edges with a Hybrid Structure Score of 0.6476.
• The approach combines intelligent point sampling, test-time augmentations, and iterative post-processing to achieve state-of-the-art results on the HoHo 22k dataset.

Transformer-Based Roof Wireframe Reconstruction from Sparse Geometric and Semantic Evidence

Problem Definition and Dataset

Urban-scale 3D scene understanding tasks remain a focal area in computer vision, with persistent challenges arising from the need to integrate fragmented geometric evidence (from sparse Structure-from-Motion, SfM, reconstructions) with semantic cues from 2D imagery. This work targets precise 3D wireframe rooftop model reconstruction using only sparse SfM point clouds, segmentation maps (Gestalt, ADE20k), and monocular depth, reflecting the practical constraints of large-scale urban data acquisition.

The research leverages the HoHo 22k dataset, a major collection with 22,000 real-world building scenes, each comprising:

• A sparse 3D SfM point cloud
• Gestalt and ADE20k-based semantic segmentation masks
• Monocular depth maps (MoGe-2)
• Ground-truth wireframe roof models

The dataset was notably refined compared to the previous year's challenge, with improvements to camera calibration uniqueness, image coverage, depth quality, and removal of spatially misaligned or inconsistent samples, although residual misalignments were managed with curation and automated filters.

Figure 1: Example scene from the HoHo 22k dataset visualizing sparse SfM, semantic classes, and depth maps.

The primary evaluation metric is the Hybrid Structure Score (HSS), defined as the harmonic mean of vertex F1 and edge IoU, with explicit geometric tolerances.

Methodology

Preprocessing and Feature Construction

Input preparation involves dynamic subsampling of the point cloud, prioritizing points classified as Gestalt edges and vertices to maximize predictive information. Each point is associated with aggregated Gestalt/ADE20k semantic features and RGB color, resulting in 180-dimensional per-point descriptors. Points are spatially normalized.

A key innovation is fusing each point's per-view Gestalt context via a frozen autoencoder projecting segmentation masks into multi-view local features. These features (32-D, then up-projected to match the embedding dimension) are fused with the point descriptors through weighted addition, supporting richer local and semantic context before Transformer encoding.

Architecture

The backbone is a Transformer encoder-decoder, structurally inspired by DETR. Points and their fused feature vectors, enhanced by fixed positional encoding (Fourier-based), are processed by a stack of encoder layers. On the decoder side, learned query embeddings represent hypothetical wireframe edges. Decoder layers alternate self-attention (among queries) with cross-attention (between queries and encoded points), mirroring DETR-style bipartite matching for geometric entities rather than object boxes.

The network outputs two predictions per query: a class (edge/background, with semantic edge classes ignored for final scoring) and normalized 3D coordinates for edge endpoints. The approach directly generates undirected 3D edges, without an explicit intermediate graph structure.

Loss Function and Training

Supervision is based on matching predicted edges to ground-truth assignments via the Hungarian algorithm, employing a combined classification (cross-entropy) and coordinate ($L_1$) loss on matched pairs. Edge directionality is handled by symmetric endpoint ordering in the norm calculation.

Critical training details include:

• Weighted sample mining for informative points
• Data augmentations (random rotation, mirroring)
• Use of 100 queries and 23M parameters
• Regular retraining on a carefully curated subset of the HoHo 22k dataset
• AdamW optimization, batch normalization, and dropout for regularization

Post-processing

After inference, spurious and low-confidence edge predictions are filtered out (class/background and confidence thresholds). To resolve duplicated or closely located vertices (due to independent endpoint prediction per edge), predictions are merged via iterative centroid computation with careful endpoint constraints. Edge-level deduplication is also performed.

Given architectural sensitivity to coordinate frame orientation, test-time augmentation is applied, including rotations and mirrorings, with ensemble averaging over multiple checkpoints to further stabilize the output.

Quantitative and Qualitative Results

This method achieves a private test set HSS of 0.6476, second only to the challenge winner (HSS 0.6542) and outperforming both the handcrafted and learned baselines by a large margin (next best HSS 0.6060). Vertex F1 and Edge IoU scores (0.7506, 0.5779) substantiate the pipeline's geometric fidelity and robustness.

Notably, the system demonstrates reliable recovery of dominant rooftop edges across diverse, incomplete input data. However, it exhibits reduced recall on less frequently represented or smaller roof features, such as chimneys and intricate details, which may result from limited semantic evidence or the aggregation/thresholding in the point sampling and post-processing pipeline.

Figure 3: Predicted rooftop wireframes closely match ground truth for main structures but miss fine details like chimneys.

Design Analysis and Ablations

Component analysis via ablations reveals:

• Model ensembling and extensive test-time augmentation yield marginal, but consistent, improvements.
• Removal of the multi-view Gestalt feature fusion slightly degrades performance, confirming its value for capturing context beyond sparse class votes.
• Uniform point sampling, instead of weighted, is only mildly detrimental, suggesting robustness to input overparameterization if general class separation is maintained.
• The architecture's performance is stable even at reduced point cloud density (down to $\sim$1800 points per scene).

Failed interventions include the explicit encoding of Plücker lines in addition to points, which did not supplement geometric cues as anticipated, possibly due to their excessive ambiguity in segment localization. Attempts to reduce the dimensionality of class embeddings, or add more structured vertex/edge losses and direct prediction heads for post-processing, did not boost HSS on the blind test set and were therefore pruned from the final submission.

Figure 4: Example of ground-truth misalignment with SfM point cloud, a key data curation challenge.

Theoretical and Practical Implications

The outlined architecture confirms that DETR-style direct set prediction, augmented by rich semantic-geometric fusion, is viable for high-fidelity structural reconstruction in challenging 3D settings with weak and fragmented evidence. The reliance on intelligent sampling, multi-view context aggregation, and rigorous post-processing rectifies many common pitfalls in set prediction applied to wireframe models: duplicate vertex/edge proposals, missed details from sparse input, and coordinate sensitivity.

Practically, the approach points toward scalable, automatic urban 3D reconstruction from low-footprint capture data—especially relevant where overhead imagery is unavailable or semantically limited. It also demonstrates transformer architectures' flexibility in direct 3D edge-graph regression.

Theoretically, the primary limitation remains the network's dependence on explicit positional encoding, requiring TTA for geometric invariance, and its incomplete capture of fine roof details. Potential future research directions include integration of graph neural priors, formal geometric constraints, or explicit modeling of edge-vertex connectivity within the Transformer framework to further improve topological precision.

Conclusion

This study presents a transformer-based approach for 3D rooftop wireframe reconstruction from sparse SfM and semantic evidence, validated on the competitive HoHo 22k benchmark. Through a combination of semantic priority sampling, multi-view context fusion, and a DETR-style encoder-decoder, the model achieves state-of-the-art wireframe reconstruction fidelity. While robust on the dominant roof structures, further improvements in fine detail recovery and rotational invariance remain open. Integrations with GNN-based regularization or generative priors present compelling avenues for refinement and broader application in scene-centric geometric deep learning.