S23DR Challenge 2026

• S23DR Challenge 2026 is a benchmark for structured 3D roof reconstruction that predicts a wireframe graph from incomplete geometric and semantic inputs with an HSS of 0.654.
• The challenge fuses sparse SfM point clouds, fitted depth maps, and semantic segmentations to create a semantic point cloud for robust 3D inference.
• The winning solution employs a two-stage DiT transformer with priority-aware sampling and 16-sample consensus, outperforming previous methods with top leaderboard performance.

Searching arXiv for the S23DR 2026 and related S23DR 2025 papers to ground the article in the cited literature. S23DR Challenge 2026 is a benchmark for structured 3D reconstruction of building roofs from incomplete geometric and semantic inputs. The task is to predict a 3D wireframe $G=(V,E)$ whose vertices are metric 3D roof keypoints and whose edges encode straight roof-line connections, while preserving both topology and metric accuracy. In the available technical record, the challenge is characterized through the winning solution, which formulates wireframe prediction as conditional set generation over vertex slots and reports the top private-leaderboard result of HSS $= 0.654$ (Skvrna et al., 4 Jun 2026).

1. Challenge formulation

The challenge targets structured 3D reconstruction of building roofs from incomplete geometric and semantic information. For each scene, corresponding to a single residential building, the required prediction is a 3D wireframe

$G=(V,E),$

where $V=\{v_i\in\mathbb{R}^3\}$ are metric 3D vertices corresponding to roof apexes, eave or ridge endpoints, building corners, and related structural keypoints, and $E\subseteq V\times V$ are straight edges connecting these vertices. Each edge carries a semantic class determined by its adjacent faces, including categories such as rake, eave, flashing, and ground line (Skvrna et al., 4 Jun 2026).

The challenge departs from image-centric formulations by explicitly not providing RGB images. Instead, reconstruction must proceed from fused geometric and semantic evidence. This makes the benchmark a problem in direct 3D structural inference rather than 2D detection followed by lifting. A plausible implication is that the task emphasizes robustness to missing observations, calibration irregularities, and sparse sampling more strongly than conventional multi-view reconstruction pipelines.

The core technical requirement is not merely to place plausible vertices in 3D space, but to recover a graph whose cardinality, geometry, and connectivity jointly match the ground-truth roof structure. This dual emphasis is reflected in the official evaluation metric, which combines a vertex-detection term with an edge-overlap term (Skvrna et al., 4 Jun 2026).

2. Inputs, outputs, and evaluation metric

The input modalities for each scene are threefold. First, the challenge provides a sparse SfM point cloud (COLMAP) containing approximately $40{,}000$ 3D points together with camera poses and intrinsics. These points are described as noisy and sparse, particularly on textureless roof surfaces. Second, it provides fitted depth maps, i.e. per-view metric depth scale-fitted against COLMAP; back-projected depth pixels yield additional 3D points, but depth frequently drifts at discontinuities and is often absent. The reported missingness is substantial: approximately 42% of training and 69% of validation views have no depth. Third, it provides semantic segmentations per view from both Gestalt roof-specific classes and ADE20K general classes, which are projected onto 3D points to produce semantic labels for COLMAP and depth points (Skvrna et al., 4 Jun 2026).

These heterogeneous sources are fused into a semantic point cloud. Each point carries normalized 3D position $xyz$, a point-type indicator distinguishing COLMAP, depth, and camera-center points, Gestalt top-1 and top-2 labels, an ADE label, and confidence scores for both geometry and semantics. The normalization is defined by centering at the median COLMAP location and scaling by the 95th-percentile radius (Skvrna et al., 4 Jun 2026).

The output representation uses a fixed-capacity graph parameterization. The model predicts 64 vertex slots, each associated with a 3D coordinate and an implicit validity flag, together with a dense matrix of pairwise edge logits. Thus the predicted graph has variable effective size but fixed representational capacity: $v_i=(x_i,y_i,z_i)\in\mathbb{R}^3,$ with validity determining whether slot $i$ corresponds to a real vertex, and edge logits determining connectivity between slot pairs (Skvrna et al., 4 Jun 2026).

Evaluation uses the Hybrid Structure Score (HSS), defined as the harmonic mean of a vertex term and an edge term. Vertices are matched with the Hungarian algorithm under a metric threshold $r$, producing vertex precision, recall, and $= 0.654$0. Edges are rendered as cylinders in 3D, and the edge term is the volumetric IoU between the predicted and ground-truth unions of cylinders. The metric is

$= 0.654$1

This metric is permutation-invariant at the vertex level and jointly sensitive to vertex existence, vertex placement, connectivity, and geometric alignment of edges (Skvrna et al., 4 Jun 2026).

3. Dataset characteristics and representational constraints

The challenge dataset is described as HoHo $= 0.654$2 hoho22k_2026_trainval and contains approximately 22k training and 200 validation buildings. Each building has 4–22 views with mean 9, building bounding boxes of 20–27 m diagonal, and wireframes containing 6–36 vertices with mean approximately 22 (Skvrna et al., 4 Jun 2026).

These statistics explain several architectural decisions in the reported winning system. The choice of 64 vertex slots is explicitly justified as being larger than the maximum typical vertex count while remaining small enough to permit dense $= 0.654$3 pairwise edge modeling inside a transformer. The model handles variable graph size via null slots and a learned validity head. This yields a conditional-set formulation in which permutation invariance is enforced by Hungarian assignment during training rather than by imposing an arbitrary canonical ordering (Skvrna et al., 4 Jun 2026).

The scene representation is also constrained by scale. Stage 1 processes up to 8192 points, while Stage 2 processes up to 16384 points after a hull-based crop. Both stages encode the scene into 1024 scene tokens, which then condition the vertex generator. This makes the benchmark simultaneously a problem in structural prediction and in controlled compression of large, heterogeneous point sets into a latent representation suitable for graph synthesis (Skvrna et al., 4 Jun 2026).

A plausible implication is that S23DR 2026 rewards methods that can preserve roof-specific semantic cues under aggressive subsampling. The winning method addresses this explicitly through priority-aware sampling that favors semantically important roof classes during token formation.

4. Winning solution architecture

The winning solution treats the wireframe as a conditional set and “denoises 64 vertex tokens with a flow-matching DiT conditioned on Perceiver-style scene tokens” (Skvrna et al., 4 Jun 2026). The pipeline has five stages: semantic point-cloud construction and normalization; Perceiver-style scene encoding; vertex-set generation by a DiT-like denoiser; a two-pass global and hull-cropped refinement scheme; and a final multi-sample consensus step.

The scene encoder first lifts each point to a $= 0.654$4-dimensional token using normalized position, Fourier features, point type, Gestalt labels, ADE labels, and confidence scalars. It then applies two full self-attention layers over all points, followed by anchored pooling into 1024 learned pooling tokens whose anchors are chosen by priority-aware Gestalt FPS. Two further self-attention layers over these pooled tokens produce the final 1024 scene tokens (Skvrna et al., 4 Jun 2026).

The vertex generator is a DiT-style transformer operating on

$= 0.654$5

the stack of 64 noisy vertex coordinates at flow time $= 0.654$6. Each slot is embedded by concatenating raw coordinates with Fourier features and projecting to model dimension $= 0.654$7. The denoiser predicts three outputs from these tokens: a velocity field $= 0.654$8 for ODE evolution, a validity logit per slot, and pairwise edge logits. The flow uses the linear interpolant

$= 0.654$9

where $G=(V,E),$0 is random initialization and $G=(V,E),$1 is the slot-assigned ground-truth vertex configuration. The target velocity is $G=(V,E),$2, and training uses a SmoothL1 flow loss (Skvrna et al., 4 Jun 2026).

The architecture is explicitly two-stage. Stage 1 performs a global pass over the entire scene to obtain a coarse wireframe and its convex hull. Stage 2 computes an inflated convex hull around the Stage-1 vertices, crops the point cloud to that region, and reruns the same architectural template on denser local evidence, initializing the 64 slots from the Stage-1 prediction rather than from random noise. The stage-2 crop is designed to remove clutter while retaining the relevant roof structure (Skvrna et al., 4 Jun 2026).

The final system adds a 16-sample consensus mechanism. Because initialization is random, multiple trajectories can yield slightly different wireframes. The method therefore runs 16 stochastic trajectories, defines a distance between predictions based on Hungarian-aligned vertex distances and edge disagreement, selects the medoid wireframe, and replaces each medoid vertex by a consensus aggregate over matched vertices from the remaining samples. This is explicitly described as a device to keep the stochastic sampler well behaved (Skvrna et al., 4 Jun 2026).

5. Training regime, inference procedure, and empirical performance

The training loss is

$G=(V,E),$3

where $G=(V,E),$4 is a SmoothL1 velocity loss, $G=(V,E),$5 is a SmoothL1 endpoint loss on final vertex positions, $G=(V,E),$6 is focal loss on slot validity, and $G=(V,E),$7 is binary cross-entropy on pairwise edge logits. Null slots are retained during training with reduced weight (Skvrna et al., 4 Jun 2026).

Stage 1 uses a 34M-parameter model with scene encoder width 320, 10 attention heads, 1024 scene tokens, a denoiser of 12 DiT blocks, and 64 slots. It is trained for 2000 epochs on 4 $G=(V,E),$8 H200 GPUs with Distributed Data Parallel, per-GPU batch size 32 for a global batch of 128, bf16 autocast, fused AdamW, weight decay $G=(V,E),$9, gradient clipping at 1.0, and a learning-rate schedule consisting of 1000-step linear warmup followed by cosine decay to a peak learning rate of $V=\{v_i\in\mathbb{R}^3\}$0. Stage 2 uses the same architecture, is trained first from scratch on hull-cropped scenes and then fine-tuned for an additional 1000 epochs at learning rate $V=\{v_i\in\mathbb{R}^3\}$1 (Skvrna et al., 4 Jun 2026).

Inference integrates the learned ODE with 50 Euler steps from $V=\{v_i\in\mathbb{R}^3\}$2 to $V=\{v_i\in\mathbb{R}^3\}$3 in both stages. In local validation ablations, Stage 1 alone achieved HSS $V=\{v_i\in\mathbb{R}^3\}$4 with total runtime 3.14 s; Stage 1 plus Stage 2 achieved HSS $V=\{v_i\in\mathbb{R}^3\}$5 with total runtime 4.30 s; and the 16-sample ensemble achieved HSS $V=\{v_i\in\mathbb{R}^3\}$6 with total runtime 5.52 s. The reported interpretation is that Stage 2 adds approximately 0.0213 HSS over Stage 1 alone, while the ensemble adds a further approximately 0.0048 HSS and reduces sensitivity to random initialization (Skvrna et al., 4 Jun 2026).

On the official competition evaluation, the submitted system ranked 1st on the private leaderboard with HSS $V=\{v_i\in\mathbb{R}^3\}$7, ahead of the second-place entry at 0.648, the learned organizer baseline at 0.474, and the handcrafted baseline at 0.391. It also achieved the highest reported vertex F1 $V=\{v_i\in\mathbb{R}^3\}$8 among all submissions. The paper interprets this as evidence of strong recovery of vertex positions and counts, with edge and topology modeling still leaving room for improvement because HSS remains below vertex F1 (Skvrna et al., 4 Jun 2026).

6. Relation to S23DR 2025 and emerging research directions

The 2026 winning system represents a substantial methodological shift relative to the S23DR Challenge 2025 winning approach. The 2025 system operated directly in 3D but used a staged discriminative pipeline: Gestalt-driven vertex candidate generation, an Extended PointNet for vertex refinement and classification on 4 m $V=\{v_i\in\mathbb{R}^3\}$9 4 m $E\subseteq V\times V$0 4 m cubic patches, and a second PointNet-like model for edge classification on radius-1 m cylinders around candidate vertex pairs. That system achieved HSS $E\subseteq V\times V$1 on the private leaderboard (Skvrna et al., 19 Jun 2025).

The comparison highlights several continuities. Both systems are explicitly 3D-centric, both avoid fragile 2D-to-3D lifting, and both rely on fused geometric and semantic evidence derived from COLMAP, depth, and semantic segmentation. However, the 2026 winner replaces explicit candidate enumeration and pairwise classification with a conditional-set generative model that predicts the entire wireframe jointly. This suggests a transition from local patch classification to global graph synthesis as the dominant paradigm in the benchmark.

The 2026 paper also identifies likely failure modes, though not exhaustively. These include missing fine structures, edge connectivity errors, occasional semantic misalignment, and instability across runs without consensus. It further suggests several directions for improvement: more explicit graph priors, better treatment of edge semantics, adaptive slot counts or hierarchical representations for more complex roofs, and extensions of the Perceiver-style fusion architecture to additional modalities such as LiDAR, multispectral data, or temporal sequences (Skvrna et al., 4 Jun 2026).

In broader methodological terms, S23DR 2026 situates roof-wireframe reconstruction at the intersection of set prediction, flow-based generative modeling in metric 3D space, and heterogeneous point-cloud encoding. The challenge, as documented by the winning system, therefore serves not only as an evaluation task for roof reconstruction, but also as a concrete testbed for permutation-invariant structured prediction under severe geometric sparsity, missing depth, and semantically noisy evidence (Skvrna et al., 4 Jun 2026).