Papers
Topics
Authors
Recent
Search
2000 character limit reached

PolyLayout: Multi-room Manhattan Layout Estimation

Published 4 Aug 2026 in cs.CV | (2608.03323v1)

Abstract: Estimating room layouts from multi-view imagery is a core task for indoor scene understanding. Existing methods are typically limited either by poor generalization to new datasets or restrictive geometric assumptions of the room shape or camera configuration. Most also estimate rooms independently, failing to exploit shared building structure such as dominant directions, ground plane or ceiling height. We propose PolyLayout, a multi-room layout estimation method that parameterizes room layouts as Manhattan 3D polygons and optimizes them jointly across multiple rooms. The optimization objective is predicted by a neural network on top of robust pre-trained visual features and trained end-to-end with supervision only on output room layouts. At the same time, camera projection and polygon updates remain explicit and model-based. This separation between learned scoring and geometry improves generalization to new datasets and camera parameters. During optimization, PolyLayout adaptively refines the polygon topology through iterative wall split and merge operations while jointly utilizing structural cues across rooms. We introduce two new multi-view multi-room layout benchmarks by providing layout annotations to existing datasets, and experiments show that PolyLayout outperforms prior approaches, both in terms of accuracy and robustness. Project page: https://ghanning.github.io/PolyLayout

Summary

  • The paper introduces a multi-view method that replaces fixed cuboids with flexible Manhattan polygons, jointly estimating room orientation, shared heights, and wall topology through differentiable featuremetric optimization.
  • PolyLayout achieves 94.3 IoU and 0.12 m Chamfer distance on ASE, and 87.4 IoU and 0.20 m Chamfer distance on ScanNet++, substantially outperforming cuboid and point-cloud baselines in several settings.
  • The method improves reconstruction of non-cuboid, multi-room interiors but requires posed images and room assignments, assumes Manhattan geometry, and trades higher accuracy for slower inference of roughly 3–5 seconds per scene.

PolyLayout: Multi-room Manhattan Layout Estimation

Research Problem and Positioning

“PolyLayout: Multi-room Manhattan Layout Estimation” (2608.03323) addresses 3D indoor layout reconstruction from multiple posed perspective images. The target representation consists of the floor, ceiling, and wall geometry of one or more rooms. Although room-layout estimation has been extensively studied from single perspective images and panoramas, those settings remain geometrically ambiguous and commonly impose strong shape restrictions. In particular, cuboid-based methods simplify optimization but cannot represent alcoves, corridors, L-shaped rooms, and other non-rectangular configurations. Conversely, methods that infer geometry through generic 3D reconstruction or point-cloud processing often exhibit substantial domain sensitivity and depend on an intermediate reconstruction whose quality varies with visual coverage.

PolyLayout builds on the featuremetric optimization paradigm of PixCuboid [(2608.03323); see also the cited predecessor in the paper], but replaces the cuboid parameterization with Manhattan polygons and extends optimization from individual rooms to complete multi-room scenes. The method combines learned image-space evidence with explicit camera projection, differentiable feature warping, vanishing-point geometry, and Levenberg–Marquardt (LM) refinement. Its central claim is that learning the scoring functions while retaining an explicit geometric optimization layer yields better cross-dataset generalization than fully learned or point-cloud-dependent alternatives.

The paper makes four principal contributions. First, it introduces a multi-view estimator for arbitrary Manhattan room polygons. Second, it jointly estimates room orientation and, when appropriate, floor and ceiling height across multiple rooms. Third, it proposes adaptive polygon topology updates through wall simplification and splitting. Fourth, it creates multi-room layout benchmarks from Aria Synthetic Environments (ASE) and manually annotated ScanNet++ v2 scenes.

Geometric Representation

Each room is represented by a Manhattan polygon embedded in 3D. The parameterization contains a global-to-local rotation and a vector of plane offsets. In the local coordinate frame, two offsets define the floor and ceiling, while the remaining offsets define alternating vertical wall planes aligned with the local xx and yy axes. Consequently, every adjacent wall pair is orthogonal, but the number of walls is not fixed. A four-wall instance reduces to a cuboid, whereas larger even numbers of wall planes represent more general orthogonal polygons.

This representation occupies a useful middle ground between rigid cuboids and unconstrained meshes. It preserves strong structural priors—vertical walls, Manhattan orientation, planar surfaces, and a horizontal floor-ceiling system—while permitting non-convex or otherwise non-cuboid floor plans. The assumption is nevertheless substantive: rooms with non-Manhattan wall orientations, curved boundaries, sloped ceilings, split-level floors, or strong geometric irregularities are outside the model class.

Initialization is derived from camera poses rather than image semantics alone. The method computes a concave hull of camera centers in an estimated horizontal frame, buffers the hull, rasterizes it on a Manhattan grid, and traces the resulting outline to obtain the initial polygon. Floor and ceiling offsets are selected from the camera distribution. The initial orientation is subsequently refined using vanishing-point alignment. This initialization is important because the optimization is local and topology-dependent; the ablation results show that concave-hull initialization substantially outperforms cuboid and circular alternatives.

Figure 1

Figure 1: Manhattan polygon parameterization and initialization from camera positions using a concave hull, buffering, and rasterization.

Learned Evidence and Model-Based Optimization

PolyLayout uses a DINOv2 ViT-S/14 encoder with two convolutional decoder heads. The first predicts dense feature maps and feature confidence, while the second predicts edge maps and edge confidence. The architecture produces a three-level image pyramid at resolutions corresponding to $1/16$, $1/4$, and $1/1$ of the input image. The encoder is fine-tuned rather than frozen, although it uses a lower learning rate than the decoder heads.

The optimization minimizes a composite objective consisting of four terms:

  • Featuremetric alignment: dense features sampled from one image are warped into another through the current 3D polygon and compared using a robust loss. Confidence maps determine the contribution of individual samples.
  • Edge alignment: points sampled along polygon edges are projected into each view and scored against learned edge maps.
  • Vanishing-point consistency: image line segments are associated with the three Manhattan vanishing points induced by the current orientation.
  • Perimeter regularization: a complexity penalty discourages unstable expansion of poorly observed polygon regions.

The featuremetric component is the principal learned signal. Unlike direct point-cloud fitting, it does not require an explicit 3D point cloud and can use multi-view photometric structure even when dense reconstruction would be unreliable. The edge term supplies sharper geometric localization, while the vanishing-point term stabilizes the global orientation. The perimeter term introduces a controlled bias toward compact layouts, particularly when some walls are not visible.

Optimization proceeds coarse-to-fine through successive LM steps. After each update, the polygon is simplified, self-intersections are removed, and camera containment is enforced. At the coarse and medium scales, long walls are split iteratively, subject to a maximum of 32 planes and a wall-width threshold of 1 m. Conversely, converged adjacent walls may be removed using a Manhattan-compatible variant of Visvalingam–Whyatt simplification. This mechanism allows the representation to change topology during inference rather than requiring the number of walls to be predicted in advance.

The network is trained end-to-end through the unrolled optimization process. Supervision is applied to the optimized layouts at each scale using ground-truth image-to-3D correspondences on walls, floors, and ceilings. The procedure therefore does not directly supervise feature descriptors as semantic embeddings; instead, it trains them to produce useful residuals for the downstream geometric optimizer. This is a significant design choice because the learned representation is evaluated by its effect on optimization rather than by an isolated feature objective.

Figure 2

Figure 2: PolyLayout jointly estimates multiple room layouts while sharing global orientation and, when valid, floor and ceiling height.

Multi-room Joint Estimation

The multi-room formulation is the paper’s principal geometric extension. Rooms in the same building share a Manhattan orientation, and the method can additionally share floor and ceiling heights. Wall offsets remain room-specific. This parameter sharing reduces the dimensionality of the joint optimization and couples otherwise underconstrained rooms through common structural variables.

The ablations support this design. On ASE, independent room optimization obtains an IoU of 93.5, whereas sharing orientation increases IoU to 93.8 and sharing orientation together with floor and ceiling height increases it to 94.3. Chamfer distance improves from 0.13 m to 0.12 m, while depth RMSE decreases from 0.11 m to 0.09 m. The gains are moderate in aggregate but consistent with the intended mechanism: shared variables receive evidence from multiple rooms and views, improving conditioning and reducing orientation drift.

The formulation assumes that image-to-room assignments are known. This assumption simplifies the joint optimization and prevents the method from addressing room segmentation or cross-room image association. In practical deployments, this preprocessing requirement may be nontrivial, especially in open-plan spaces, through-door views, or scenes with ambiguous room boundaries.

Dataset Construction and Evaluation Protocol

The authors construct two multi-room evaluation resources. ASE provides synthetic scenes with known camera trajectories and ground-truth floor plans. One hundred scenes are sampled for validation and another hundred for testing, with five image sets of ten views per room. Because uniform sampling can select highly redundant views, the authors use a visibility-based heuristic that favors images covering previously unseen surface points and penalizes repeated visibility.

For ScanNet++ v2, the authors manually annotate room layouts in 80 scenes. The annotations include both cuboid and more general layouts, although rooms are restricted to flat walls, a single horizontal ceiling, and a planar floor. Three sets of ten DSLR images per room are sampled. The evaluation includes 3D IoU, Chamfer distance, wall recall, room recall, depth RMSE, normal recall, and inference time. Wall recall measures whether sampled points on each ground-truth wall are captured by the prediction within 0.25 m; room recall requires all walls of a room to satisfy this criterion.

The evaluation is appropriately broader than IoU alone. IoU can remain high despite missing short walls or producing locally incorrect topology, whereas wall and room recall explicitly measure structural completeness. The paper’s results also illustrate why these metrics matter: high volumetric overlap does not necessarily imply that all architectural boundaries have been recovered.

Quantitative Results

PolyLayout performs particularly strongly on ASE. It achieves an IoU of 94.3, a Chamfer distance of 0.12 m, wall recall of 89.0, room recall of 79.0, depth RMSE of 0.09 m, and normal recall of 98.2. Its performance substantially exceeds PixCuboid, which obtains an IoU of 68.1 and a Chamfer distance of 0.93 m. It also greatly outperforms Plane-DUSt3R and RoomFormer under the image-only or sparse-reconstruction conditions used in the experiment.

On ScanNet++ v2, PolyLayout obtains an IoU of 87.4, Chamfer distance of 0.20 m, wall recall of 69.3, room recall of 36.3, depth RMSE of 0.16 m, and normal recall of 91.6. PixCuboid achieves an IoU of 78.8 and Chamfer distance of 0.36 m. The lower recall values on ScanNet++ reflect more detailed geometry, shorter wall segments, incomplete scans, and deviations from the shared-height assumptions.

The results on 2D-3D-Semantics are more nuanced. On cuboid rooms, PolyLayout reaches an IoU of 90.0 and Chamfer distance of 0.15 m, marginally improving on PixCuboid’s IoU of 89.0 and Chamfer distance of 0.18 m. However, PixCuboid has higher wall recall and room recall—93.8 and 85.0 compared with PolyLayout’s 92.2 and 80.6—and is faster, requiring 0.42 s rather than 1.11 s. This is an important qualification: PolyLayout does not dominate a cuboid-specialized method on every metric or dataset. Its advantage is primarily its broader geometric model and stronger performance on non-cuboid, multi-room, and cross-domain settings.

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3: Qualitative comparison of room-layout predictions on ScanNet++ showing the relative robustness of PolyLayout to non-cuboid geometry.

On ASE, PolyLayout requires 5.48 s per scene, compared with 1.01 s for PixCuboid and 0.06 s for RoomFormer. On ScanNet++, it requires 3.54 s, compared with 0.87 s for PixCuboid. The additional cost comes from feature extraction, multi-room optimization, line detection, and dynamic topology handling. Thus, the method trades latency for geometric accuracy and generality.

Backbone and Objective Ablations

The DINOv2 backbone contributes substantially, but the results show that the gains are not attributable to the backbone alone. On ASE, PolyLayout with ResNet-101 achieves an IoU of 82.7 and Chamfer distance of 0.43 m, while PolyLayout with DINOv2 reaches 94.3 and 0.12 m. On ScanNet++, the corresponding improvement is from 84.7 to 87.4 IoU and from 0.26 m to 0.20 m Chamfer distance.

The cross-method comparison is especially informative. PolyLayout with ResNet-101 outperforms PixCuboid with DINOv2 on both ASE and ScanNet++, indicating that polygonal representation, multi-room coupling, topology adaptation, and the full objective are responsible for a substantial fraction of the improvement. On 2D-3D-Semantics, however, PixCuboid with DINOv2 achieves the best results, consistent with its specialization for cuboid rooms.

The featuremetric term alone is insufficient. With ResNet-101, the feature-only configuration obtains an IoU of 24.6 on ASE; DINOv2 improves this to 45.1 but remains far below the complete system. Adding edge, vanishing-point, and perimeter terms raises the result to 94.3. A particularly strong comparison concerns point-cloud baselines: on cuboid ScanNet++ scenes, RANSAC fitting obtains an IoU of 26.1, and robust point-to-cuboid distance fitting obtains 45.9. Featuremetric alignment alone reaches 63.5, while combining it with edge alignment reaches 92.4 IoU, 0.15 m Chamfer distance, and 88.1 room recall. These results demonstrate that learned image-space evidence is not merely an alternative implementation of geometric point fitting.

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4: DINOv2 produces more discriminative feature and edge evidence than the ResNet-based baseline, particularly on synthetic ASE imagery.

The perimeter regularizer improves volumetric and pixel-wise accuracy but slightly decreases recall. This trade-off is theoretically expected: shrinking unobserved regions can reduce geometric error while causing some small or weakly supported walls to disappear. Similarly, disabling simplification produces overly complex polygons and degrades performance, confirming that topology control is not a cosmetic post-processing step.

Robustness to Camera-Pose Quality

Although PolyLayout assumes posed images, the authors evaluate the effect of replacing ground-truth poses with predictions from π3\pi^3. After alignment for evaluation, the predicted-pose setting obtains an IoU of 89.4, Chamfer distance of 0.21 m, wall recall of 84.4, room recall of 73.2, depth RMSE of 0.15 m, and normal recall of 97.0. Relative to ground-truth poses—94.3 IoU, 0.12 m Chamfer, and 89.0 wall recall—the degradation is measurable but not catastrophic.

This experiment supports the claim that the method can potentially operate with standard SfM or learned pose estimators. It does not, however, constitute a fully unposed evaluation: the predicted poses are aligned with the ground truth before measuring the final layouts. A deployment-oriented assessment would need to include scale, global-frame, correspondence, and room-assignment uncertainty without post hoc alignment.

Failure Modes and Scope of Validity

PolyLayout struggles when complex room sections are not observed in any input view or when the visible evidence is insufficient to distinguish multiple plausible wall configurations. The visibility-based sampling strategy mitigates redundant views but cannot recover completely unobserved geometry. The adaptive polygon mechanism can also produce incorrect topology when a long wall should be split but the visual evidence is weak, or when simplification removes a short wall that is structurally important.

Figure 5

Figure 5

Figure 5

Figure 5

Figure 5: Failure cases in ASE involving missed walls and incorrect reconstruction of complex room shapes under incomplete visual coverage.

The Manhattan prior creates a second class of limitations. ScanNet++ includes rooms that are not strictly Manhattan or do not share a common ceiling height. The authors handle these cases by restricting annotations and selectively disabling height sharing, but the results necessarily measure performance on a filtered subset of the full indoor-scene distribution. Extending the representation to Atlanta-world or piecewise-Manhattan structures would increase applicability, although it would also enlarge the optimization space and complicate parameter sharing.

Another limitation is computational. PolyLayout is considerably slower than lightweight direct predictors and specialized cuboid optimizers. Its iterative LM procedure, multi-scale feature warping, line detection, and topology updates are well suited to accuracy-oriented reconstruction but less directly suited to high-frame-rate robotics or interactive mobile AR. A practical system may therefore require a fast learned initializer followed by selective geometric refinement.

Implications for Indoor AI Systems

The paper has practical implications for robotics, AR, spatial mapping, and embodied AI. A reliable multi-room polygonal representation can provide a compact structural map for navigation, collision checking, object placement, view planning, and scene-level reasoning. Compared with dense meshes, polygons are computationally economical and directly expose architectural primitives. Compared with cuboids, they preserve sufficient topological flexibility for realistic floor plans.

The method also illustrates a broader systems principle: explicit geometric optimization remains valuable when the target structure is low-dimensional and strongly constrained, even when the image evidence is learned. End-to-end training of the feature extractor and optimizer allows the network to specialize for the downstream residual landscape, while the model-based layer enforces projection consistency and valid geometric structure. This division can improve data efficiency and interpretability relative to predicting all layout parameters through a single feed-forward network.

The multi-room parameter-sharing mechanism is particularly relevant to future spatial foundation models. Shared orientation and height are weak but informative building-level constraints. More extensive sharing—such as aligned wall segments, door locations, room adjacency, or structural level hypotheses—could connect local layout estimation to global scene graphs. However, such extensions would require uncertainty-aware association because incorrect sharing can propagate errors across rooms.

Future Directions

Several developments follow naturally from the method. First, joint estimation of camera poses, image-room assignments, and layouts would remove two important external assumptions. Differentiable pose refinement could be integrated into the existing LM framework, although gauge freedoms and local minima would need explicit treatment.

Second, the polygon model could be generalized beyond strict Manhattan geometry. A mixture of Manhattan frames, Atlanta-world walls, or piecewise-planar boundary primitives could accommodate older buildings and irregular interiors while retaining a structured optimization space.

Third, the learned confidence maps could be calibrated probabilistically. Current confidence weighting is useful for suppressing unreliable image regions, but calibrated uncertainty could support robust fusion across rooms, active view selection, and principled stopping criteria for topology updates.

Fourth, inference could be accelerated through amortized initialization, fewer LM iterations, sparse feature sampling, or learned proposal mechanisms for wall splits and merges. The current method demonstrates that iterative refinement is effective, but its 3–5 s scene-level runtime leaves room for substantial engineering and algorithmic improvement.

Finally, the introduced benchmarks should be expanded with richer annotations for doors, openings, wall thickness, non-Manhattan structures, and temporal sequences. Such data would enable evaluation of complete architectural reconstruction rather than only closed room volumes.

Conclusion

“PolyLayout: Multi-room Manhattan Layout Estimation” (2608.03323) presents a technically coherent integration of learned featuremetric alignment and explicit geometric optimization. Its main advance is not simply replacing cuboids with polygons, but combining flexible Manhattan topology, multi-room parameter sharing, adaptive wall refinement, and DINOv2-derived visual evidence within a differentiable LM framework. The strongest results occur on ASE and ScanNet++, where PolyLayout substantially improves over cuboid-based, point-cloud-based, and generic learned baselines. At the same time, the weaker recall and runtime on selected settings, dependence on known poses and room assignments, and sensitivity to unobserved or non-Manhattan geometry define clear boundaries of applicability. The work supports continued development of hybrid learned-model-based systems for structured 3D scene understanding.

Whiteboard

Explain it Like I'm 14

1. What is the paper about?

The paper introduces PolyLayout, a computer-vision system that creates a 3D map of indoor rooms from several photographs.

The system tries to find:

  • Where the walls are
  • Where the floor and ceiling are
  • The shape of each room
  • How several rooms fit together

This is useful for robots, virtual reality, augmented reality, and systems that need to understand buildings.

Many older systems assume that every room is a simple rectangular box. PolyLayout is more flexible: it can describe rooms with extra corners, such as an L-shaped room.

2. Main research questions

The researchers wanted to find out:

  1. Can a computer estimate room shapes from multiple ordinary camera images?
  2. Can it handle rooms that are not simple rectangles?
  3. Can it estimate several rooms together instead of treating each room separately?
  4. Does sharing information between rooms make the predictions more accurate?
  5. Can the method work well on both computer-generated images and real photographs?

The main idea is that rooms in the same building often share important features. For example, their walls usually point in the same main directions, and they may have the same floor and ceiling height.

3. How does PolyLayout work?

PolyLayout combines a trained neural network with traditional geometry and optimization.

Representing a room as a polygon

A polygon is a flat shape made from straight sides. A rectangle is a polygon with four sides, while an L-shaped room has more sides.

PolyLayout represents a room as a 3D version of a polygon:

  • The floor and ceiling are horizontal planes.
  • The walls are vertical.
  • Walls meet at right angles.

This is called a Manhattan layout, similar to the straight lines and right angles found in many buildings. The system can use four walls for a box-shaped room or add more walls for more complicated shapes.

Starting with the camera positions

The method knows the positions and directions of the cameras that took the photographs. It uses these positions to make an initial guess about the room’s shape.

This is like drawing a rough fence around all the places where the cameras were standing. The system then turns this rough outline into a room-shaped polygon.

A neural network studies the images

A neural network examines each image and produces several kinds of information:

  • Feature maps: descriptions of what different parts of the image look like
  • Edge maps: guesses about where important lines, such as wall edges, are located
  • Confidence maps: estimates of how reliable each guess is

The network uses a model called DINOv2, which is a type of vision transformer. In everyday terms, it is a powerful image-understanding system that looks at relationships between many parts of an image, rather than only examining nearby pixels.

Improving the room shape step by step

After making an initial guess, PolyLayout repeatedly changes the walls to make the layout better match the photographs. This process is called optimization.

The system checks several types of evidence:

  1. Matching image features: The same wall should look consistent when seen from different cameras.
  2. Matching image edges: Projected room edges should line up with lines detected in the photographs.
  3. Vanishing points: Parallel lines in the real world often appear to meet at a point in an image. For example, the edges of a hallway may seem to meet far away. These points help estimate the room’s directions.
  4. Reasonable shape: The system discourages room outlines from becoming unnecessarily complicated or strangely large.

PolyLayout works from rough information to detailed information. This is called coarse-to-fine optimization. It is similar to first drawing a picture with a large pencil, then adding details with a smaller pencil.

Adding and removing walls

One important feature is that the number of walls is not fixed.

During the process, PolyLayout can:

  • Remove walls that do not seem necessary
  • Split a long wall into two walls
  • Correct walls that cross each other
  • Expand walls if a camera would otherwise end up outside the room

This allows the method to adjust to different room shapes automatically.

Estimating several rooms together

For a building with multiple rooms, PolyLayout can optimize all the rooms at the same time.

The rooms can share:

  • The same main orientation
  • The same floor height
  • The same ceiling height

This is like solving several connected puzzles together instead of solving each one alone. Information from one room can help make the answers for the other rooms more accurate.

4. Data and experiments

The researchers tested PolyLayout on three sources of data:

Aria Synthetic Environments

This is a computer-generated collection of indoor scenes. It contains many rooms, including both rectangular and more complicated rooms.

The researchers created new validation and test sets, each containing 100 scenes. They selected images that gave good views of different parts of the rooms.

ScanNet++

This contains real indoor scans. The researchers manually marked the floors, ceilings, and walls in 80 scenes to create room-layout examples for testing.

These scenes included both simple rectangular rooms and more complicated shapes.

2D-3D-Semantics

This dataset contains rooms that are mostly rectangular. It was used to compare PolyLayout with methods designed specifically for box-shaped rooms.

5. How was success measured?

The researchers used several measurements.

  • IoU: Measures how much the predicted room overlaps the correct room. A higher number is better.
  • Chamfer distance: Measures how far the predicted surfaces are from the correct surfaces. A lower distance is better.
  • Wall recall: Measures how many real walls the system successfully finds.
  • Room recall: Measures how many rooms have all of their walls correctly found.
  • Depth error: Measures how accurately the system predicts distances in the scene. A smaller error is better.
  • Normal accuracy: Measures whether surfaces point in the correct direction.

These measurements are like grading a student’s map by checking both its overall shape and the accuracy of individual walls.

6. Main findings

The results show that PolyLayout performed especially well on the more general room-layout tasks.

For example, using the DINOv2 network, PolyLayout achieved the following results:

Dataset 3D overlap, IoU Surface distance Wall recall Room recall
Aria Synthetic Environments 94.3% 0.12 m 89.0% 79.0%
ScanNet++ 87.4% 0.20 m 69.3% 36.3%
2D-3D-Semantics 90.0% 0.15 m 92.2% 80.6%

The method was strongest on the synthetic Aria data. It also worked well on ScanNet++, although real scenes are more difficult because of incomplete scans, hidden walls, furniture, and unusual room shapes.

The researchers also compared PolyLayout with PixCuboid, an earlier method. PixCuboid assumes every room is a cuboid, meaning a box with six rectangular surfaces. PolyLayout was generally better on Aria and ScanNet++, where rooms can have more complicated shapes.

On the mostly rectangular 2D-3D-Semantics dataset, PixCuboid performed slightly better. This makes sense because PixCuboid was specially designed for box-shaped rooms, while PolyLayout is designed to handle a wider range of shapes.

The comparison also showed that the improvement did not come only from replacing the old ResNet network with DINOv2. PolyLayout using the older ResNet still often performed better than PixCuboid using DINOv2. This suggests that the whole design of PolyLayout—especially its flexible polygons and joint multi-room optimization—is important.

7. Limitations

PolyLayout can still fail when:

  • Parts of a room are not visible in any photograph
  • The room has a very complicated shape
  • Walls are hidden by objects
  • The real room does not follow the right-angle Manhattan assumption
  • The camera images do not cover the room well

The method also assumes that the system already knows the camera positions and which images belong to which room. Therefore, it does not solve every part of the indoor-mapping problem by itself.

8. Why is this research important?

PolyLayout makes room reconstruction more useful because it is not limited to simple rectangular rooms. It can create more realistic 3D layouts and can use shared information across several rooms.

In the future, this could help:

  • Robots navigate buildings
  • Augmented-reality systems understand walls and floors
  • Virtual-reality programs create digital copies of real places
  • Architects and designers make floor plans from photographs
  • Computer systems build 3D models of homes, offices, and other indoor spaces

Overall, the paper shows that combining machine learning with clear geometric rules can produce accurate room maps. The neural network learns what walls and edges look like, while the geometry makes sure the final answer forms a sensible 3D building layout.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

  • Dependence on known camera calibration and poses: The method assumes known camera intrinsics, extrinsics, and image-to-room assignments; its robustness to pose noise, inaccurate calibration, unposed images, or automatic room association is not evaluated.
  • Restricted geometric model: PolyLayout assumes Manhattan layouts with vertical walls, orthogonal wall directions, a flat floor, and a single horizontal ceiling per room. Its performance on non-Manhattan, slanted, curved, stepped, multi-level, or irregular ceiling geometries remains unresolved.
  • Unvalidated shared-height assumption: Joint optimization can share floor and ceiling heights across rooms, but ScanNet++ contains scenes where this assumption does not hold. The paper does not establish a principled method for detecting when height sharing is invalid or quantify the resulting failure modes.
  • Unexplored sharing of wall parameters: The paper notes that wall locations could also be shared across rooms but does not investigate this potentially useful structural constraint, including how to identify corresponding or physically shared walls.
  • Reliance on sufficient visual coverage: Complex rooms may fail when parts of walls are not visible in any input image. The method does not provide uncertainty-aware completion, active view selection, or a systematic analysis of the minimum camera number and coverage required for reliable reconstruction.
  • Limited evaluation of camera sampling strategies: Visibility-based sampling is designed specifically for ASE, whereas uniform sampling is used for ScanNet++. The relative effects of sampling policy, number of views, baseline, viewing direction, and occlusion are not isolated through controlled experiments.
  • Sensitivity to manually selected hyperparameters: Performance may depend on fixed choices such as the buffer distance, rasterization resolution, wall-simplification threshold, maximum number of planes, sampling exponent, VP threshold, and LM iteration count. Their sensitivity and transferability across scene scales and datasets are not comprehensively studied.
  • Topology optimization lacks formal guarantees: Wall splitting, merging, simplification, and self-intersection removal are heuristic operations. The paper does not characterize when these operations recover the correct topology, whether they can permanently eliminate necessary walls, or whether the optimization is stable under different initialization conditions.
  • Initialization failures are insufficiently characterized: The initialization depends on the camera-position concave hull, a fixed buffer, and an estimated vertical direction. The method’s behavior when cameras cover only a small region, lie near room boundaries, have poor orientation diversity, or are unevenly distributed is not systematically evaluated.
  • Potential scale dependence of geometric thresholds: Several thresholds are specified in meters, such as the 3 m buffer, 1 m wall-splitting criterion, 0.2 m cuboid-fitting tolerance, and 0.25 m evaluation spacing. It remains unclear how the approach performs in unusually small or large rooms and whether scale-normalized settings are needed.
  • No robustness analysis for inaccurate or missing image observations: The experiments do not appear to test blur, exposure changes, motion, dynamic objects, severe occlusions, missing views, or substantial changes in image quality and camera resolution.
  • Limited domain-diversity assessment: The evaluation uses ASE, ScanNet++, and 2D-3D-Semantics, but the generalization claim is not tested on other building types, cultural or architectural styles, sensors, lighting conditions, or real-world video sequences.
  • Synthetic-to-real transfer remains only partially explained: DINOv2 improves qualitative generalization to ASE, but the paper does not determine which components—pre-training, feature representation, training mixture, image sampling, or optimization—are responsible for the transfer advantage.
  • Small manually annotated Manhattan training set: The training data include only 107 manually annotated Manhattan layouts in addition to 391 cuboid rooms. The effect of annotation quantity, annotation noise, and layout-shape diversity on performance is not quantified.
  • Annotation quality and ambiguity are not analyzed: ScanNet++ ground truth is produced by manually selecting mesh vertices and fitting planes, and only rooms satisfying specific structural conditions are retained. Inter-annotator variation, fitting uncertainty, and the impact of imperfect or incomplete scans are not reported.
  • Dataset selection may bias the evaluation: ScanNet++ scenes with incomplete lidar scans, non-flat walls, or unsuitable ceilings are excluded, so performance on the broader distribution of real indoor environments remains unknown.
  • Metric sensitivity is unresolved: Wall recall, room recall, IoU, Chamfer distance, depth RMSE, and normal recall may reward different types of errors. The paper does not analyze whether these metrics adequately capture topological correctness, room usability, doorway connectivity, or practical downstream utility.
  • Scene-level averaging may obscure room-level failures: Metrics for ASE and ScanNet++ are computed per scene rather than per room. The consequences of this choice, especially for scenes containing many rooms or one severely failed room, are not examined.
  • Door and window geometry is not reconstructed: Although doors and windows are used in ASE image filtering and ground-truth processing, the predicted output represents walls, floor, and ceiling rather than openings. The effect of doors and windows on layout estimation and the possibility of jointly estimating them remain open.
  • No explicit handling of occluding furniture or movable objects: The approach relies on image features, edges, and geometric consistency but does not separately model furniture, clutter, mirrors, glass, or other objects that can produce misleading structural cues.
  • Confidence maps are not calibrated: Confidence predictions weight the featuremetric and edge costs, but their calibration, reliability under domain shift, and relationship to layout uncertainty are not evaluated.
  • Output uncertainty is not provided: The method returns a single optimized layout. It does not quantify ambiguity, produce alternative hypotheses, or identify which walls, heights, or orientations are weakly constrained by the available views.
  • End-to-end training through discrete topology changes is incompletely understood: Polygon simplification, splitting, and self-intersection removal are not fully differentiable, yet the network is trained through unrolled optimization. The paper does not clarify how gradients interact with these discrete operations or how training behaves near topology transitions.
  • Training and inference objectives differ: The VP and perimeter costs are disabled during training but used during inference, and the perimeter term is further disabled at the finest scale. The effect of this objective mismatch and of each inference-only term is not fully isolated.
  • Computational scalability is unclear: The method jointly optimizes multiple rooms and uses multi-scale feature warping and LM steps, but runtime and memory scaling with the number of rooms, images, polygon walls, and sampled points are not comprehensively reported.
  • Dependence on external detectors and libraries: Performance relies on DeepLSD for line segments, DINOv2 features, and Shapely-based geometry processing. Failure propagation from these components and alternatives for environments where they are unavailable or unreliable are not studied.
  • The best input-view configuration is unknown: Experiments use fixed sets of 10 images per room, with multiple predefined sampled sets. Accuracy as a function of one, two, or many views—and the point at which additional views cease to help—is not established.
  • Camera trajectories with severe temporal correlation are not addressed: The sampling procedure is intended to avoid redundant views, but the method’s performance on ordinary video streams containing highly correlated frames is not evaluated.
  • Joint optimization across rooms assumes a common building frame: The method shares global orientation across rooms, but it is unclear how it handles buildings with rotated wings, non-orthogonal room arrangements, or local coordinate frames that violate a single global Manhattan orientation.
  • Failure recovery is limited: The paper documents failures in complex and partially unobserved rooms but does not propose mechanisms for detecting failed optimization, restarting from alternative hypotheses, or combining predictions from multiple initializations.
  • Downstream usefulness is not demonstrated: Although applications in robotics, augmented reality, and scene understanding are motivated, the paper does not evaluate whether the reconstructed layouts improve navigation, localization, mapping, AR occlusion, or other downstream tasks.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Indoor mapping for robotics and autonomous systems — Robotics, logistics, service automation PolyLayout can convert multiple posed RGB images into Manhattan-aligned 3D room polygons containing walls, floors, and ceilings. This provides a lightweight structural map for robot navigation, obstacle-free route planning, localization, and task planning in homes, offices, warehouses, and public buildings. Potential workflow: camera/visual-inertial SLAM → camera poses and images → PolyLayout → navigable floor plan or collision geometry. Dependencies: reliable camera poses and intrinsics; sufficient visual coverage; environments that are approximately Manhattan, with vertical walls and flat floors/ceilings. Complex, occluded, or unobserved walls can cause failures.
  • Rapid floor-plan generation from handheld or wearable cameras — Construction, real estate, facilities management The predicted 3D polygons can be projected onto the ground plane to produce 2D floor plans. This could support rapid documentation of existing buildings using image sequences from smartphones, body-worn cameras, or head-mounted devices. Potential products: a mobile “walk-through-to-floor-plan” application, preliminary property measurement software, or facility inventory tools. Dependencies: the current method requires known camera poses, so a practical product would need integration with visual-inertial SLAM or another pose-estimation system. Outputs should be treated as preliminary measurements rather than legally certified plans.
  • AR/VR spatial anchoring and occlusion — Augmented reality, virtual reality, gaming Multi-room polygons can supply approximate wall, floor, and ceiling geometry for placing virtual objects, generating physically plausible occlusion masks, and maintaining spatial anchors across rooms. Joint optimization of room orientation and floor/ceiling height is particularly useful for maintaining a consistent coordinate frame across a building. Potential tools: automatic room-boundary detection in AR headsets, indoor scene setup for VR, and room-scale game environments. Dependencies: low-latency inference and accurate headset tracking are required for real-time use. Manhattan assumptions may be inadequate for curved, angled, or irregular architecture.
  • 3D reconstruction assistance for surveying and digital twins — Architecture, engineering, construction, property management PolyLayout can provide an initial structural model from imagery that is subsequently refined with RGB-D scans, lidar, CAD information, or manual edits. Its explicit geometric representation is suitable for generating wall planes and room boundaries rather than only producing opaque neural predictions. Potential workflow: image capture → automatic layout proposal → engineer or surveyor validation → BIM/digital-twin export. Dependencies: the model estimates room boundaries, not complete BIM semantics such as doors, windows, electrical systems, or material properties. Human verification remains necessary for high-stakes applications.
  • Scene understanding for indoor analytics — Smart buildings, security, facility operations The layout can provide a geometric coordinate system for aggregating observations across multiple cameras and rooms. Applications include room occupancy analysis, camera coverage planning, indoor asset tracking, and spatial event indexing. Dependencies: image-to-room assignment is assumed for multi-room scenes, and privacy, data-retention, and surveillance regulations may limit deployment. Occluded or unseen walls reduce the reliability of the spatial index.
  • Simulation environment generation — Robotics research, computer graphics, synthetic-data generation Estimated Manhattan polygons can be converted into simple collision meshes or simulator maps for testing navigation and embodied-AI systems. The approach is especially useful when a full point cloud is unavailable but posed images exist. Potential tools: automated scene importers for robotics simulators, synthetic sensor-rendering pipelines, and benchmark-generation utilities. Dependencies: additional object-level reconstruction is required for realistic interaction; the paper’s method primarily models structural surfaces.
  • Academic benchmarking and reproducible research — Computer vision, robotics, spatial AI The newly annotated Aria Synthetic Environments and ScanNet++ multi-view, multi-room benchmarks can support evaluation of layout estimation, multi-room reconstruction, camera coverage, and generalization across synthetic and real domains. The reported metrics—3D IoU, Chamfer distance, wall recall, room recall, depth, and normal accuracy—enable more detailed comparison than a single aggregate score. Dependencies: practical impact depends on release of the annotations, image sets, code, and standardized evaluation protocols. Dataset bias remains a concern because the annotations focus on rooms with flat walls and single horizontal ceilings.
  • Human-in-the-loop building documentation — Government, education, maintenance, emergency planning Municipalities, universities, and facility operators could use PolyLayout to create initial room maps for asset inventories, maintenance planning, accessibility audits, or emergency-response preparation. The system’s confidence maps can help identify views or wall segments that require manual inspection. Dependencies: outputs must be validated before use in safety-critical contexts. Incomplete scans, unusual geometry, and image privacy constraints may limit coverage.

Long-Term Applications

  • Large-scale automated digital twins of buildings — AEC, real estate, smart infrastructure A scaled system could reconstruct multi-room layouts across entire buildings and combine them with semantic detectors for doors, windows, furniture, utilities, and occupancy. Shared orientation and floor/ceiling parameters provide a useful basis for globally consistent building models. Required development: joint optimization across floors and disconnected spaces, automatic image-room assignment, robust handling of non-Manhattan structures, uncertainty estimation, and export to BIM standards such as IFC. Dependencies: large, diverse training datasets and robust pose estimation in poorly textured or repetitive interiors.
  • Autonomous exploration and active perception — Robotics, drones, inspection The visibility-based image-sampling strategy could be extended into an active exploration policy. A robot could use preliminary layout estimates and confidence maps to select the next camera position that maximizes coverage of uncertain or unseen walls. Potential workflow: estimate layout → identify low-confidence regions → plan camera motion → capture additional views → update layout. Dependencies: closed-loop uncertainty calibration, real-time optimization, safe navigation, and methods for handling moving people or furniture.
  • Self-correcting SLAM and camera-pose refinement — Localization, mapping, AR Because PolyLayout explicitly models planes, vanishing points, feature consistency, and camera projections, future versions could jointly optimize room geometry and camera poses rather than assuming poses are known. This could reduce drift in long indoor trajectories and improve map consistency. Dependencies: joint geometry-pose optimization is more ambiguous and computationally demanding. It would require robust initialization and safeguards against incorrect Manhattan alignment.
  • Building-scale navigation and accessibility planning — Public policy, healthcare, eldercare, transport Building-wide layout reconstruction could support route planning for wheelchairs, evacuation analysis, indoor wayfinding, and accessibility assessment. Combined with semantic recognition, systems could identify narrow passages, inaccessible thresholds, or room connectivity. Dependencies: the current output does not explicitly model doors, stairs, ramps, elevators, or temporary obstructions. Regulatory or emergency use would require certified accuracy and comprehensive validation.
  • Construction progress and renovation monitoring — Construction, insurance, infrastructure management Repeated image captures could be compared over time to detect changes in wall placement, room subdivision, or structural completion. PolyLayout’s polygon representation is well suited to geometric differencing and progress dashboards. Dependencies: consistent camera calibration and pose alignment across capture sessions; the method must distinguish genuine construction changes from furniture, occlusions, lighting changes, and reconstruction errors.
  • Physics-aware simulation and embodied-AI training — Robotics, AI, simulation More accurate layout estimates could seed interactive simulators in which robots learn navigation, manipulation, and multi-room task execution. The explicit wall, floor, and ceiling geometry could be combined with inferred materials, objects, and affordances. Dependencies: current layouts are structurally simplified and do not capture complete scene semantics, physical properties, or dynamic agents. Sim-to-real transfer would require validation on diverse real environments.
  • Generalized reconstruction beyond Manhattan rooms — Computer vision, architecture, heritage preservation Extending the method to Atlanta-world, curved, slanted, multi-level, or non-planar interiors would broaden its use to historical buildings, industrial facilities, atria, and modern architectures. The existing separation between learned image scoring and explicit geometry provides a foundation for replacing or expanding the polygon parameterization. Dependencies: new geometric models, training annotations, topology operations, and stronger handling of self-occlusions are required. The paper reports that complex shapes and unobserved scene regions remain important failure modes.
  • Uncertainty-aware professional measurement systems — Surveying, insurance, finance, policy The predicted feature and edge confidence maps could eventually be converted into calibrated uncertainty estimates for wall locations, room dimensions, and missing regions. Such estimates would allow systems to distinguish automatic measurements suitable for planning from areas requiring human inspection. Dependencies: the paper uses confidence maps primarily as optimization weights, not as validated probabilistic uncertainty. Calibration, error guarantees, and standardized reporting would be necessary for insurance, lending, or regulatory workflows.
  • Privacy-preserving indoor mapping — Healthcare, workplaces, consumer technology A future deployment could retain only structural polygons and discard raw imagery after processing, reducing exposure of personal or sensitive visual information. This may enable spatial documentation in hospitals, offices, and homes while limiting long-term storage of identifiable images. Dependencies: image processing may still expose sensitive data during capture or inference, and derived maps can themselves reveal confidential building layouts. Strong access controls, on-device inference, and privacy impact assessments would be required.
View Paper Prompt  View All Prompts 

Glossary

  • α\alpha-shape: A geometric shape constructed from a set of points using a tunable parameter to control concavity. “an α\alpha-shape \cite{edelsbrunner2003shape} is computed around the camera positions”
  • Ablation experiment: An experiment that evaluates the contribution of individual components by removing or changing them. “PolyLayout is evaluated on both synthetic and real data and we validate the design in a number of ablation experiments.”
  • Atlanta world: A scene model in which walls are vertical but need not be mutually orthogonal. “The Atlanta world \cite{schindler2004atlanta} differs from the Manhattan world in that the walls are vertical but not necessarily orthogonal”
  • Backbone: The principal feature-extraction network underlying a larger neural-network architecture. “Impact of Network Backbone”
  • Bilinear interpolation: An interpolation method that estimates values between pixels using a weighted average of the four neighboring pixels. “The images in the resulting pyramid are resized with bilinear interpolation”
  • Chamfer distance: A measure of similarity between two point sets based on the average nearest-neighbor distance between them. “We adopt the 3D (IoU, Chamfer distance)”
  • Coarse-to-fine optimization: An optimization strategy that begins with low-resolution representations and progressively uses finer resolutions. “It is initialized from the camera poses (\cref{subsec:room-layout-initialization}) and then refined through a coarse-to-fine optimization”
  • Confidence map: A spatial map whose values indicate the estimated reliability of predictions at different image locations. “One decoder outputs the feature map anditsassociatedconfidencemapand its associated confidence map. The other predicts the edge map alongwithconfidencealong with confidence.”
  • Concave hull: A boundary enclosing points that can follow inward curves rather than only forming a convex boundary. “we first compute the concave hull”
  • Convolutional decoder: A neural-network component that transforms encoded features into spatial prediction maps, often by upsampling them. “two convolutional decoders (heads)”
  • Coplanarity: The geometric property of multiple points or objects lying on the same plane. “Checks for coplanarity have been omitted for brevity.”
  • Cuboid: A three-dimensional shape bounded by six rectangular faces, with opposite faces parallel. “It randomly samples points (without replacement) from the input point cloud and constructs the cuboid plane-by-plane.”
  • DINOv2: A self-supervised vision-transformer model used to produce robust visual representations. “We also suggest a new network architecture compared to PixCuboid, with a DINOv2 \cite{oquab2023dinov2} encoder”
  • Depth RMSE: Root mean squared error computed between predicted and reference depth values. “pixel-wise (depth RMSE, normal angle recall at 10\degree)”
  • Dense feature map: A spatially organized array of learned feature vectors, typically containing one vector for each image location. “to predict a dense feature map ∈RW×H×D\in\mathbb{R}^{W\times H\times D}”
  • End-to-end training: Training in which the parameters of all relevant model components are optimized jointly according to the final task objective. “The network is trained end-to-end”
  • Featuremetric alignment: Alignment based on the similarity of learned feature representations rather than directly on image intensities. “PixCuboid is an optimization-based approach to layout estimation centered around featuremetric alignment.”
  • Featuremetric cost: An objective measuring the inconsistency between learned features after geometric warping between views. “the featuremetric cost”
  • Foundation model: A large, broadly pretrained model that can be adapted to many downstream tasks. “Plane-DUSt3R retrains the foundation model DUSt3R”
  • Generalized loss function: A robust loss formulation designed to accommodate different error distributions and reduce the influence of outliers. “ρ\rho is the generalized loss function from \cite{barron2019general}”
  • Ground truth: Reference data regarded as the correct answer for training or evaluation. “Ground truth layouts are shown in \textcolor{GTColor}{\bf blue}.”
  • Holistic scene understanding: Comprehensive interpretation of a scene, including its geometry, objects, structure, and relationships. “with applications in robotics, augmented reality and holistic scene understanding.”
  • Inlier: A data point that agrees with a fitted model within a specified error tolerance. “pick the cuboid with the maximum number of inliers, defined as points that are within 0.2 m of its faces.”
  • Indoor world model: A geometric model representing an indoor environment with aligned walls and a shared floor and ceiling structure. “By adding a single-floor, single-ceiling restriction we get the indoor world model”
  • Intrinsics: Camera parameters describing internal projection properties such as focal length and principal point. “with known intrinsics Ki\bm{K}_i”
  • IoU (intersection over union): A ratio measuring the overlap between predicted and reference regions or volumes. “We adopt the 3D (IoU, Chamfer distance)”
  • Levenberg–Marquardt optimization: A nonlinear least-squares optimization method that combines gradient-descent and Gauss–Newton updates. “Our method is based on optimizing an initial layout through successive Levenberg-Marquardt \cite{levenberg1944method,marquardt1963algorithm} steps”
  • Manhattan frame: A coordinate frame whose axes correspond to the dominant mutually orthogonal directions of a scene. “The Manhattan frame is estimated during optimization.”
  • Manhattan polygon: A polygon whose edges are aligned with a set of mutually orthogonal coordinate axes. “room layouts are represented by Manhattan polygons.”
  • Manhattan world: A scene model in which structural planes are aligned with three principal orthogonal axes. “The Manhattan world \cite{coughlan1999manhattan}, where planes are aligned with three principal axes”
  • Mean prediction time: The average time required by a method to generate a prediction. “but compute them per scene instead of per room for ASE and ScanNet++.”
  • Multi-view geometry: The geometric analysis of a scene using observations from multiple camera views. “Layout estimation from multiple views is a less studied area of research.”
  • Normal vector: A vector perpendicular to a surface or plane. “ni\bm{n}_i is the normal vector of the surface”
  • Normal-angle recall: The fraction of surface-normal predictions whose angular error is below a specified threshold. “normal angle recall at 10\degree”
  • Perimeter cost: An optimization penalty that discourages excessively complex or expanding polygon layouts. “We introduce a perimeter cost EperE_{per} to penalize complex polygon layouts”
  • Perspective view: An image formed by a camera projection in which apparent size changes with depth. “PixCuboid leverage multiple perspective views for the task.”
  • Photometric cue: Visual information derived from image brightness, color, or intensity. “reconstruct indoor scenes from video sequences by combining geometric and photometric cues.”
  • Plane offset: A scalar specifying the position of a plane relative to a coordinate origin along its normal direction. “a plane offset vector d=[d1d2⋯dp]∈Rp\bm{d} = \begin{bmatrix} d_1 & d_2 & \cdots & d_p \end{bmatrix} \in \mathbb{R}^p”
  • Point cloud: A collection of three-dimensional points representing the sampled geometry of a scene. “Our method PolyLayout does not require point clouds”
  • Pose: The position and orientation of a camera in a scene. “Given posed images”
  • Principal axes: The mutually orthogonal axes defining the dominant directions of a geometric environment. “planes are aligned with three principal axes”
  • Probability map: A spatial map assigning sampling probabilities to image locations. “To create the probability map κ^\kappa for point sampling”
  • Rasterization: The conversion of geometric shapes into a grid of pixels or cells. “The shape is converted to a Manhattan polygon by rasterizing it”
  • Recall: The proportion of relevant reference instances that are successfully detected or reconstructed. “In addition, we define wall and room recall as follows.”
  • Residual block: A neural-network module that adds a block’s transformed output to its input through a skip connection. “We apply layer normalization \cite{ba2016layer} to the input of the residual blocks”
  • Robust loss function: A loss function designed to limit the effect of outliers on optimization. “ρ\rho is a robust loss function”
  • RANSAC: A randomized algorithm that repeatedly fits a model to samples and selects the model supported by the most inliers. “We run the algorithm in a RANSAC loop”
  • Self-occlusion: Occlusion in which one part of a scene or object blocks another part from the same scene or object. “it is possible to have self-occlusions with polygons”
  • Self-intersection: A geometric condition in which a polygon boundary crosses itself. “Any self-intersections that might have been introduced by the LM step or simplification are removed”
  • Semantic mesh: A three-dimensional mesh whose geometry is associated with semantic scene information. “the provided semantic mesh (DmeshD_{mesh})”
  • SfM (structure from motion): A computer-vision technique that jointly estimates camera motion and three-dimensional structure from multiple images. “Direct alignment of deep features has been used in a number of areas such as point cloud registration \cite{huang2020feature}, camera localization \cite{sarlin2021back} and SfM”
  • SLAM (simultaneous localization and mapping): A method for estimating a sensor’s position while constructing a map of an unknown environment. “a point cloud obtained from a visual-inertial SLAM system”
  • Sub-pixel interpolation: Interpolation used to estimate an image or feature value at a non-integer pixel location. “[⋅][\cdot] represents lookup with sub-pixel interpolation.”
  • Topology: The connectivity and structural arrangement of geometric elements such as polygon edges and vertices. “PolyLayout adaptively refines the polygon topology”
  • Transformer: A neural-network architecture based primarily on attention mechanisms for modeling relationships among input elements. “Pintore \etal \cite{pintore20183d} take as input multiple panoramic views and can reconstruct multi-room scenes.”
  • Unrolled optimization: The representation of iterative optimization steps as differentiable computational layers in a neural network. “propagate the gradients back through the unrolled optimization process.”
  • Vanishing point: The image location toward which projections of parallel three-dimensional lines converge. “Thanks to our Manhattan assumption the polygon P\mathcal{P} has only three vanishing points”
  • ViT (vision transformer): A transformer-based neural network that processes images as sequences of patches. “having a DINOv2 \cite{oquab2023dinov2} ViT encoder”
  • Visibility-based sampling: A sampling strategy that selects views according to how much previously unseen scene content they observe. “we employ a visibility-based sampling scheme that tries to maximize the shared visual coverage.”
  • Warping: The geometric transformation of image or feature data from one view into another using a scene model. “where Wi→j\mathcal{W}_{i\to j} denotes the warping to image j_j via the polygon P\mathcal{P}.”
  • Weighted MSE loss: A mean squared error objective in which different errors receive different weights. “we also pre-train the edge maps with a weighted MSE loss”
  • Vision transformer encoder: An image encoder that uses transformer layers to generate learned visual representations from image patches. “We use a pre-trained ViT-S/14 model without registers for the encoder.”
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Tweets

Sign up for free to view the 2 tweets with 527 likes about this paper.

Sign Up for Free