S23DR 2026 Winning Solution

Abstract: This text presents the winning solution to the S23DR 2026 challenge for structured 3D wireframe reconstruction from sparse SfM, fitted depth, and semantic segmentations. The method treats vertices as a conditional set and denoises 64 vertex tokens with a flow-matching DiT conditioned on Perceiver-style scene tokens. A global pass predicts the coarse structure, a hull-cropped second pass refines it, and a small multi-sample consensus step keeps the stochastic sampler well behaved. The final system ranked first on the private leaderboard, achievingHSS = 0.654.

• The paper presents a two-stage denoising pipeline that refines noisy vertex predictions into a structurally coherent 3D wireframe.
• It employs a Perceiver-like scene encoder and DiT-style transformer with flow-matching ODE supervision, achieving a 0.654 Hybrid Structure Score and 0.791 vertex F1.
• The methodology leverages ensemble consensus and enriched semantic point clouds, enhancing geometric precision and robustness against sparse inputs.

Structured 3D Wireframe Reconstruction: The S23DR 2026 Winning Solution

Problem Formulation and Challenge Structure

The S23DR 2026 challenge targets the reconstruction of 3D wireframes representing building geometries from non-standard, highly sparse inputs—COLMAP point clouds, pose-aligned depth maps, and dense semantic segmentations. The goal is to predict a labeled 3D graph $G=(V,E)$, where the vertex set $V$ denotes semantically meaningful building features (such as roof apexes and corners), and the edge set $E$ encodes straight-line geometry and semantic attachment between the predicted vertices. The challenge metric, Hybrid Structure Score (HSS), harmonizes permutation-invariant vertex matching and 3D edge-based volumetric IoU, penalizing both topological inconsistencies and geometric imprecision.

The dataset (HoHo), comprising over 22,000 training and 200 validation residential scenes, is intentionally noisy and incomplete. RGB images are withheld for privacy, complicating the task versus prior works and pushing the limits of geometric and semantic set prediction on sparse, multimodal data.

Methodological Framework

The proposed solution models wireframe prediction as conditional set generation in $\mathbb{R}^3$, eschewing traditional direct 2D-to-3D lifting. The pipeline is decomposed into two key stages coupled with a consensus-based ensemble.

Architectural Overview

An enriched semantic point cloud forms the input, where each point is encoded with normalized coordinates, Fourier features, point type embeddings (COLMAP, depth, or camera center), semantic (Gestalt and ADE20k) embeddings, and confidence scalars. This comprehensive representation is critical for capturing both the metric structure and the semantic priors associated with roof architecture.

The processing pipeline is as follows:

• Scene Encoder: A Perceiver-like attention-driven encoder projects the enriched points into high-dimensional context tokens. It incorporates two full self-attention layers for global context aggregation, and 1024 pooling tokens, smartly anchored via Gestalt-informed FPS, for efficient information distillation and locality preservation.
• Vertex Denoiser: The generative core is a DiT-style transformer operating over 64 vertex tokens. Utilizing flow-matching ODE supervision, the model predicts velocity vectors, validity logits (to discriminate active/inactive slots), and pairwise edge logits. All are causally conditioned on scene tokens and adaptive time embeddings.

The denoising trajectory of this system is visualized as vertex tokens iteratively refine their positions, transforming a noisy initialization into a structurally and semantically coherent wireframe. The stages are clearly observed when visualizing the coarse Stage 1 and the subsequent hull-cropped refinement in Stage 2, as shown below.

(Figure 1)

Figure 1: Denoising trajectory from sparse input to final wireframe via two-stage vertex refinement and learned flow.

Two-Stage Inference and Ensemble Consensus

• Stage 1 (Global Generation): Operates on a globally subsampled (8192 points) enriched point cloud. It outputs a coarse but topologically accurate initial vertex set.
• Stage 2 (Hull-Cropped Refinement): The convex hull of Stage 1's output is expanded and used to crop the scene, removing extraneous clutter. Inside this hull, a denser point cloud (16384 points) allows the second denoiser pass to focus on metric precision and fine-grained structure.
• Ensemble Consensus: During inference, 16 stochastic denoising trajectories are run in parallel, leveraging variations in slot initialization. The medoid, computed over Hungarian-aligned vertex distances and edge agreement, serves as the final prediction skeleton, with further consensus refinement enhancing geometric robustness and reducing stochastic variance.

Experimental Results and Benchmarking

Training leveraged large model configurations ($d=320$, 12-block denoisers, 34M parameters), distributed across 4×H200 GPUs with bf16 autocast, DDP, and advanced optimization protocols (fused AdamW, gradient clipping, cosine LR decay). Stage 1 and Stage 2 were trained sequentially, with Stage 2 initialization strictly derived from Stage 1 predictions.

Key numerical results are as follows:

• The final system achieved a Hybrid Structure Score (HSS) of 0.654 on the private leaderboard, surpassing both the second-place competitor (0.648) and dramatically outperforming the learned (0.474) and handcrafted (0.391) organizer baselines.
• The system attained a vertex F1 of 0.791, marking highest geometric precision among all submissions.
• The 16-sample ensemble further stabilized performance, providing modest but consistent gains over single-run inference and dramatically reducing susceptibility to random initialization effects.

Implications and Theoretical Significance

The methodology signals a marked advancement in conditional set modeling for 3D structure prediction. By leveraging flow-matching for permutation-invariant set prediction and harnessing a hybrid of metric and semantic conditioning in the scene encoder, the system demonstrates resilience to extreme input sparsity and noise, a scenario where legacy 2D-lifting and classical MVS approaches systematically underperform.

Practically, the two-stage generator with consensus refinement sets a template for industrial-scale 3D modeling pipelines, especially in applications—urban modeling, GIS, AR—that must operate under data privacy restrictions and without dense imagery. The theoretical underpinnings further highlight the effectiveness of flow-based generative modeling in permutation-invariant graph prediction tasks, and the architectural progression from global topology to local refinement presents a robust pattern applicable to other sparse-to-structure mapping domains.

Future Directions

Open directions include the extension of set-based conditional generation to handle increased graph complexity, integration with neural radiance field guidance where permitted, incorporation of additional uncertainty quantification in inference, and adaptation to unsupervised or weakly-supervised domains. The demonstrated efficacy of two-stage denoising with semantic-enriched context is also a natural fit for complex indoor scenes and multiscale object layouts beyond architectural wireframes.

Conclusion

The S23DR 2026 winning solution establishes a robust and high-precision architecture for 3D wireframe reconstruction under highly constrained input regimes. Its two-stage, permutation-invariant generative design, enriched semantic modeling, and robust consensus mechanisms set a new performance baseline for the challenge and provide a promising framework for related structured prediction tasks in computer vision and spatial AI (2606.06695).