PolyLayout: Multi-room Manhattan Layout Estimation

Abstract: Estimating room layouts from multi-view imagery is a core task for indoor scene understanding. Existing methods are typically limited either by poor generalization to new datasets or restrictive geometric assumptions of the room shape or camera configuration. Most also estimate rooms independently, failing to exploit shared building structure such as dominant directions, ground plane or ceiling height. We propose PolyLayout, a multi-room layout estimation method that parameterizes room layouts as Manhattan 3D polygons and optimizes them jointly across multiple rooms. The optimization objective is predicted by a neural network on top of robust pre-trained visual features and trained end-to-end with supervision only on output room layouts. At the same time, camera projection and polygon updates remain explicit and model-based. This separation between learned scoring and geometry improves generalization to new datasets and camera parameters. During optimization, PolyLayout adaptively refines the polygon topology through iterative wall split and merge operations while jointly utilizing structural cues across rooms. We introduce two new multi-view multi-room layout benchmarks by providing layout annotations to existing datasets, and experiments show that PolyLayout outperforms prior approaches, both in terms of accuracy and robustness. Project page: https://ghanning.github.io/PolyLayout

• The paper introduces a multi-view method that replaces fixed cuboids with flexible Manhattan polygons, jointly estimating room orientation, shared heights, and wall topology through differentiable featuremetric optimization.
• PolyLayout achieves 94.3 IoU and 0.12 m Chamfer distance on ASE, and 87.4 IoU and 0.20 m Chamfer distance on ScanNet++, substantially outperforming cuboid and point-cloud baselines in several settings.
• The method improves reconstruction of non-cuboid, multi-room interiors but requires posed images and room assignments, assumes Manhattan geometry, and trades higher accuracy for slower inference of roughly 3–5 seconds per scene.

PolyLayout: Multi-room Manhattan Layout Estimation

Research Problem and Positioning

“PolyLayout: Multi-room Manhattan Layout Estimation” (2608.03323) addresses 3D indoor layout reconstruction from multiple posed perspective images. The target representation consists of the floor, ceiling, and wall geometry of one or more rooms. Although room-layout estimation has been extensively studied from single perspective images and panoramas, those settings remain geometrically ambiguous and commonly impose strong shape restrictions. In particular, cuboid-based methods simplify optimization but cannot represent alcoves, corridors, L-shaped rooms, and other non-rectangular configurations. Conversely, methods that infer geometry through generic 3D reconstruction or point-cloud processing often exhibit substantial domain sensitivity and depend on an intermediate reconstruction whose quality varies with visual coverage.

PolyLayout builds on the featuremetric optimization paradigm of PixCuboid [(2608.03323); see also the cited predecessor in the paper], but replaces the cuboid parameterization with Manhattan polygons and extends optimization from individual rooms to complete multi-room scenes. The method combines learned image-space evidence with explicit camera projection, differentiable feature warping, vanishing-point geometry, and Levenberg–Marquardt (LM) refinement. Its central claim is that learning the scoring functions while retaining an explicit geometric optimization layer yields better cross-dataset generalization than fully learned or point-cloud-dependent alternatives.

The paper makes four principal contributions. First, it introduces a multi-view estimator for arbitrary Manhattan room polygons. Second, it jointly estimates room orientation and, when appropriate, floor and ceiling height across multiple rooms. Third, it proposes adaptive polygon topology updates through wall simplification and splitting. Fourth, it creates multi-room layout benchmarks from Aria Synthetic Environments (ASE) and manually annotated ScanNet++ v2 scenes.

Geometric Representation

Each room is represented by a Manhattan polygon embedded in 3D. The parameterization contains a global-to-local rotation and a vector of plane offsets. In the local coordinate frame, two offsets define the floor and ceiling, while the remaining offsets define alternating vertical wall planes aligned with the local $x$ and $y$ axes. Consequently, every adjacent wall pair is orthogonal, but the number of walls is not fixed. A four-wall instance reduces to a cuboid, whereas larger even numbers of wall planes represent more general orthogonal polygons.

This representation occupies a useful middle ground between rigid cuboids and unconstrained meshes. It preserves strong structural priors—vertical walls, Manhattan orientation, planar surfaces, and a horizontal floor-ceiling system—while permitting non-convex or otherwise non-cuboid floor plans. The assumption is nevertheless substantive: rooms with non-Manhattan wall orientations, curved boundaries, sloped ceilings, split-level floors, or strong geometric irregularities are outside the model class.

Initialization is derived from camera poses rather than image semantics alone. The method computes a concave hull of camera centers in an estimated horizontal frame, buffers the hull, rasterizes it on a Manhattan grid, and traces the resulting outline to obtain the initial polygon. Floor and ceiling offsets are selected from the camera distribution. The initial orientation is subsequently refined using vanishing-point alignment. This initialization is important because the optimization is local and topology-dependent; the ablation results show that concave-hull initialization substantially outperforms cuboid and circular alternatives.

Figure 1: Manhattan polygon parameterization and initialization from camera positions using a concave hull, buffering, and rasterization.

Learned Evidence and Model-Based Optimization

PolyLayout uses a DINOv2 ViT-S/14 encoder with two convolutional decoder heads. The first predicts dense feature maps and feature confidence, while the second predicts edge maps and edge confidence. The architecture produces a three-level image pyramid at resolutions corresponding to $1/16$, $1/4$, and $1/1$ of the input image. The encoder is fine-tuned rather than frozen, although it uses a lower learning rate than the decoder heads.

The optimization minimizes a composite objective consisting of four terms:

• Featuremetric alignment: dense features sampled from one image are warped into another through the current 3D polygon and compared using a robust loss. Confidence maps determine the contribution of individual samples.
• Edge alignment: points sampled along polygon edges are projected into each view and scored against learned edge maps.
• Vanishing-point consistency: image line segments are associated with the three Manhattan vanishing points induced by the current orientation.
• Perimeter regularization: a complexity penalty discourages unstable expansion of poorly observed polygon regions.

The featuremetric component is the principal learned signal. Unlike direct point-cloud fitting, it does not require an explicit 3D point cloud and can use multi-view photometric structure even when dense reconstruction would be unreliable. The edge term supplies sharper geometric localization, while the vanishing-point term stabilizes the global orientation. The perimeter term introduces a controlled bias toward compact layouts, particularly when some walls are not visible.

Optimization proceeds coarse-to-fine through successive LM steps. After each update, the polygon is simplified, self-intersections are removed, and camera containment is enforced. At the coarse and medium scales, long walls are split iteratively, subject to a maximum of 32 planes and a wall-width threshold of 1 m. Conversely, converged adjacent walls may be removed using a Manhattan-compatible variant of Visvalingam–Whyatt simplification. This mechanism allows the representation to change topology during inference rather than requiring the number of walls to be predicted in advance.

The network is trained end-to-end through the unrolled optimization process. Supervision is applied to the optimized layouts at each scale using ground-truth image-to-3D correspondences on walls, floors, and ceilings. The procedure therefore does not directly supervise feature descriptors as semantic embeddings; instead, it trains them to produce useful residuals for the downstream geometric optimizer. This is a significant design choice because the learned representation is evaluated by its effect on optimization rather than by an isolated feature objective.

Figure 2: PolyLayout jointly estimates multiple room layouts while sharing global orientation and, when valid, floor and ceiling height.

Multi-room Joint Estimation

The multi-room formulation is the paper’s principal geometric extension. Rooms in the same building share a Manhattan orientation, and the method can additionally share floor and ceiling heights. Wall offsets remain room-specific. This parameter sharing reduces the dimensionality of the joint optimization and couples otherwise underconstrained rooms through common structural variables.

The ablations support this design. On ASE, independent room optimization obtains an IoU of 93.5, whereas sharing orientation increases IoU to 93.8 and sharing orientation together with floor and ceiling height increases it to 94.3. Chamfer distance improves from 0.13 m to 0.12 m, while depth RMSE decreases from 0.11 m to 0.09 m. The gains are moderate in aggregate but consistent with the intended mechanism: shared variables receive evidence from multiple rooms and views, improving conditioning and reducing orientation drift.

The formulation assumes that image-to-room assignments are known. This assumption simplifies the joint optimization and prevents the method from addressing room segmentation or cross-room image association. In practical deployments, this preprocessing requirement may be nontrivial, especially in open-plan spaces, through-door views, or scenes with ambiguous room boundaries.

Dataset Construction and Evaluation Protocol

The authors construct two multi-room evaluation resources. ASE provides synthetic scenes with known camera trajectories and ground-truth floor plans. One hundred scenes are sampled for validation and another hundred for testing, with five image sets of ten views per room. Because uniform sampling can select highly redundant views, the authors use a visibility-based heuristic that favors images covering previously unseen surface points and penalizes repeated visibility.

For ScanNet++ v2, the authors manually annotate room layouts in 80 scenes. The annotations include both cuboid and more general layouts, although rooms are restricted to flat walls, a single horizontal ceiling, and a planar floor. Three sets of ten DSLR images per room are sampled. The evaluation includes 3D IoU, Chamfer distance, wall recall, room recall, depth RMSE, normal recall, and inference time. Wall recall measures whether sampled points on each ground-truth wall are captured by the prediction within 0.25 m; room recall requires all walls of a room to satisfy this criterion.

The evaluation is appropriately broader than IoU alone. IoU can remain high despite missing short walls or producing locally incorrect topology, whereas wall and room recall explicitly measure structural completeness. The paper’s results also illustrate why these metrics matter: high volumetric overlap does not necessarily imply that all architectural boundaries have been recovered.

Quantitative Results

PolyLayout performs particularly strongly on ASE. It achieves an IoU of 94.3, a Chamfer distance of 0.12 m, wall recall of 89.0, room recall of 79.0, depth RMSE of 0.09 m, and normal recall of 98.2. Its performance substantially exceeds PixCuboid, which obtains an IoU of 68.1 and a Chamfer distance of 0.93 m. It also greatly outperforms Plane-DUSt3R and RoomFormer under the image-only or sparse-reconstruction conditions used in the experiment.

On ScanNet++ v2, PolyLayout obtains an IoU of 87.4, Chamfer distance of 0.20 m, wall recall of 69.3, room recall of 36.3, depth RMSE of 0.16 m, and normal recall of 91.6. PixCuboid achieves an IoU of 78.8 and Chamfer distance of 0.36 m. The lower recall values on ScanNet++ reflect more detailed geometry, shorter wall segments, incomplete scans, and deviations from the shared-height assumptions.

The results on 2D-3D-Semantics are more nuanced. On cuboid rooms, PolyLayout reaches an IoU of 90.0 and Chamfer distance of 0.15 m, marginally improving on PixCuboid’s IoU of 89.0 and Chamfer distance of 0.18 m. However, PixCuboid has higher wall recall and room recall—93.8 and 85.0 compared with PolyLayout’s 92.2 and 80.6—and is faster, requiring 0.42 s rather than 1.11 s. This is an important qualification: PolyLayout does not dominate a cuboid-specialized method on every metric or dataset. Its advantage is primarily its broader geometric model and stronger performance on non-cuboid, multi-room, and cross-domain settings.

Figure 3: Qualitative comparison of room-layout predictions on ScanNet++ showing the relative robustness of PolyLayout to non-cuboid geometry.

On ASE, PolyLayout requires 5.48 s per scene, compared with 1.01 s for PixCuboid and 0.06 s for RoomFormer. On ScanNet++, it requires 3.54 s, compared with 0.87 s for PixCuboid. The additional cost comes from feature extraction, multi-room optimization, line detection, and dynamic topology handling. Thus, the method trades latency for geometric accuracy and generality.

Backbone and Objective Ablations

The DINOv2 backbone contributes substantially, but the results show that the gains are not attributable to the backbone alone. On ASE, PolyLayout with ResNet-101 achieves an IoU of 82.7 and Chamfer distance of 0.43 m, while PolyLayout with DINOv2 reaches 94.3 and 0.12 m. On ScanNet++, the corresponding improvement is from 84.7 to 87.4 IoU and from 0.26 m to 0.20 m Chamfer distance.

The cross-method comparison is especially informative. PolyLayout with ResNet-101 outperforms PixCuboid with DINOv2 on both ASE and ScanNet++, indicating that polygonal representation, multi-room coupling, topology adaptation, and the full objective are responsible for a substantial fraction of the improvement. On 2D-3D-Semantics, however, PixCuboid with DINOv2 achieves the best results, consistent with its specialization for cuboid rooms.

The featuremetric term alone is insufficient. With ResNet-101, the feature-only configuration obtains an IoU of 24.6 on ASE; DINOv2 improves this to 45.1 but remains far below the complete system. Adding edge, vanishing-point, and perimeter terms raises the result to 94.3. A particularly strong comparison concerns point-cloud baselines: on cuboid ScanNet++ scenes, RANSAC fitting obtains an IoU of 26.1, and robust point-to-cuboid distance fitting obtains 45.9. Featuremetric alignment alone reaches 63.5, while combining it with edge alignment reaches 92.4 IoU, 0.15 m Chamfer distance, and 88.1 room recall. These results demonstrate that learned image-space evidence is not merely an alternative implementation of geometric point fitting.

Figure 4: DINOv2 produces more discriminative feature and edge evidence than the ResNet-based baseline, particularly on synthetic ASE imagery.

The perimeter regularizer improves volumetric and pixel-wise accuracy but slightly decreases recall. This trade-off is theoretically expected: shrinking unobserved regions can reduce geometric error while causing some small or weakly supported walls to disappear. Similarly, disabling simplification produces overly complex polygons and degrades performance, confirming that topology control is not a cosmetic post-processing step.

Robustness to Camera-Pose Quality

Although PolyLayout assumes posed images, the authors evaluate the effect of replacing ground-truth poses with predictions from $\pi^3$. After alignment for evaluation, the predicted-pose setting obtains an IoU of 89.4, Chamfer distance of 0.21 m, wall recall of 84.4, room recall of 73.2, depth RMSE of 0.15 m, and normal recall of 97.0. Relative to ground-truth poses—94.3 IoU, 0.12 m Chamfer, and 89.0 wall recall—the degradation is measurable but not catastrophic.

This experiment supports the claim that the method can potentially operate with standard SfM or learned pose estimators. It does not, however, constitute a fully unposed evaluation: the predicted poses are aligned with the ground truth before measuring the final layouts. A deployment-oriented assessment would need to include scale, global-frame, correspondence, and room-assignment uncertainty without post hoc alignment.

Failure Modes and Scope of Validity

PolyLayout struggles when complex room sections are not observed in any input view or when the visible evidence is insufficient to distinguish multiple plausible wall configurations. The visibility-based sampling strategy mitigates redundant views but cannot recover completely unobserved geometry. The adaptive polygon mechanism can also produce incorrect topology when a long wall should be split but the visual evidence is weak, or when simplification removes a short wall that is structurally important.

Figure 5: Failure cases in ASE involving missed walls and incorrect reconstruction of complex room shapes under incomplete visual coverage.

The Manhattan prior creates a second class of limitations. ScanNet++ includes rooms that are not strictly Manhattan or do not share a common ceiling height. The authors handle these cases by restricting annotations and selectively disabling height sharing, but the results necessarily measure performance on a filtered subset of the full indoor-scene distribution. Extending the representation to Atlanta-world or piecewise-Manhattan structures would increase applicability, although it would also enlarge the optimization space and complicate parameter sharing.

Another limitation is computational. PolyLayout is considerably slower than lightweight direct predictors and specialized cuboid optimizers. Its iterative LM procedure, multi-scale feature warping, line detection, and topology updates are well suited to accuracy-oriented reconstruction but less directly suited to high-frame-rate robotics or interactive mobile AR. A practical system may therefore require a fast learned initializer followed by selective geometric refinement.

Implications for Indoor AI Systems

The paper has practical implications for robotics, AR, spatial mapping, and embodied AI. A reliable multi-room polygonal representation can provide a compact structural map for navigation, collision checking, object placement, view planning, and scene-level reasoning. Compared with dense meshes, polygons are computationally economical and directly expose architectural primitives. Compared with cuboids, they preserve sufficient topological flexibility for realistic floor plans.

The method also illustrates a broader systems principle: explicit geometric optimization remains valuable when the target structure is low-dimensional and strongly constrained, even when the image evidence is learned. End-to-end training of the feature extractor and optimizer allows the network to specialize for the downstream residual landscape, while the model-based layer enforces projection consistency and valid geometric structure. This division can improve data efficiency and interpretability relative to predicting all layout parameters through a single feed-forward network.

The multi-room parameter-sharing mechanism is particularly relevant to future spatial foundation models. Shared orientation and height are weak but informative building-level constraints. More extensive sharing—such as aligned wall segments, door locations, room adjacency, or structural level hypotheses—could connect local layout estimation to global scene graphs. However, such extensions would require uncertainty-aware association because incorrect sharing can propagate errors across rooms.

Future Directions

Several developments follow naturally from the method. First, joint estimation of camera poses, image-room assignments, and layouts would remove two important external assumptions. Differentiable pose refinement could be integrated into the existing LM framework, although gauge freedoms and local minima would need explicit treatment.

Second, the polygon model could be generalized beyond strict Manhattan geometry. A mixture of Manhattan frames, Atlanta-world walls, or piecewise-planar boundary primitives could accommodate older buildings and irregular interiors while retaining a structured optimization space.

Third, the learned confidence maps could be calibrated probabilistically. Current confidence weighting is useful for suppressing unreliable image regions, but calibrated uncertainty could support robust fusion across rooms, active view selection, and principled stopping criteria for topology updates.

Fourth, inference could be accelerated through amortized initialization, fewer LM iterations, sparse feature sampling, or learned proposal mechanisms for wall splits and merges. The current method demonstrates that iterative refinement is effective, but its 3–5 s scene-level runtime leaves room for substantial engineering and algorithmic improvement.

Finally, the introduced benchmarks should be expanded with richer annotations for doors, openings, wall thickness, non-Manhattan structures, and temporal sequences. Such data would enable evaluation of complete architectural reconstruction rather than only closed room volumes.

Conclusion

“PolyLayout: Multi-room Manhattan Layout Estimation” (2608.03323) presents a technically coherent integration of learned featuremetric alignment and explicit geometric optimization. Its main advance is not simply replacing cuboids with polygons, but combining flexible Manhattan topology, multi-room parameter sharing, adaptive wall refinement, and DINOv2-derived visual evidence within a differentiable LM framework. The strongest results occur on ASE and ScanNet++, where PolyLayout substantially improves over cuboid-based, point-cloud-based, and generic learned baselines. At the same time, the weaker recall and runtime on selected settings, dependence on known poses and room assignments, and sensitivity to unobserved or non-Manhattan geometry define clear boundaries of applicability. The work supports continued development of hybrid learned-model-based systems for structured 3D scene understanding.