PixCuboid: Cuboid Room Layout Estimation

• PixCuboid is an optimization-based method that estimates a room's cuboid layout from multi-view RGB images by aligning dense deep features across views.
• It combines a U-Net style network with a Levenberg–Marquardt optimizer in a multi-scale, end-to-end training pipeline to refine geometry and enhance accuracy.
• The approach achieves significant performance improvements on benchmarks by integrating feature alignment, edge detection, and vanishing-point constraints for precise 3D reconstructions.

PixCuboid is an optimization-based method for cuboid-shaped room layout estimation from multiple posed RGB images. It models a room as a single 3D cuboid and estimates that cuboid by aligning dense deep features across views rather than regressing layout parameters in a single forward pass. The method combines a U-Net style network with a non-linear least-squares optimizer, trains the feature extractor end-to-end through unrolled optimization, and evaluates on two benchmarks with manually verified ground truth 3D cuboids derived from ScanNet++ and 2D‑3D‑Semantics (Hanning et al., 6 Aug 2025).

1. Problem formulation and scope

PixCuboid addresses coarse room layout estimation under a cuboid-room assumption. The inputs are a set of RGB images

$\{I_i\}_{i=1}^n,\quad I_i \in \mathbb{R}^{W\times H \times 3},$

together with known camera intrinsics $\mathbf{K}_i$ and extrinsics $(\mathbf{R}_i,\mathbf{t}_i)$ for each view. The output is a cuboid $\mathcal{C}$ representing the room layout in 3D (Hanning et al., 6 Aug 2025).

The paper positions room layout estimation as a source of coarse geometric cues for downstream tasks. The stated motivations include AR/VR, robotics and localization, indoor scene understanding, and 3D vision pipelines. In this formulation, the layout is intended to preserve structural components—walls, floor, and ceiling—while abstracting away clutter and furniture. Unlike methods based on a single perspective image or a single panorama, PixCuboid uses multiple posed perspective views and exploits multi-view geometry and parallax to resolve scale and room extent (Hanning et al., 6 Aug 2025).

The method assumes cuboid-shaped rooms, known camera poses, and a Manhattan-like structure for the vanishing-point prior to be meaningful. A plausible implication is that PixCuboid is designed for metric reconstruction in settings where camera calibration or SfM/SLAM poses are already available, rather than as a stand-alone single-image layout regressor.

2. Cuboid representation and geometric objective

PixCuboid represents the room as a single 3D cuboid $\mathcal{C}$ with 9 degrees of freedom. The parameterization consists of a rotation $\mathbf{R} \in SO(3)$ and six signed offsets $\mathbf{d} \in \mathbb{R}^6$, one for each face, measured along the rotated coordinate axes. In the cuboid-aligned frame, the six faces are defined by

$\begin{aligned} (1, 0, 0)\,\mathbf{X} &= d_1, \ (-1, 0, 0)\,\mathbf{X} &= d_2, \ (0, 1, 0)\,\mathbf{X} &= d_3, \ (0,-1, 0)\,\mathbf{X} &= d_4, \ (0, 0, 1)\,\mathbf{X} &= d_5, \ (0, 0,-1)\,\mathbf{X} &= d_6. \end{aligned}$

The translation of the cuboid is encoded implicitly in $\mathbf{d}$. The paper emphasizes that this parameterization is minimal, that each face is parameterized independently via its offset, and that different axis permutations of $\mathbf{R}$ with corresponding reordering of $\mathbf{K}_i$0 represent the same physical cuboid (Hanning et al., 6 Aug 2025).

With camera pose $\mathbf{K}_i$1, a 3D point $\mathbf{K}_i$2 projects to image $\mathbf{K}_i$3 as

$\mathbf{K}_i$4

PixCuboid defines a warp $\mathbf{K}_i$5 that maps an image point from view $\mathbf{K}_i$6 to view $\mathbf{K}_i$7 through the hypothesized cuboid: the point is back-projected to a ray, intersected with the cuboid, and then reprojected into the target view. This produces the featuremetric term

$\mathbf{K}_i$8

where $\mathbf{K}_i$9 are dense feature maps, $(\mathbf{R}_i,\mathbf{t}_i)$0 are confidence weights derived from predicted confidence maps, and $(\mathbf{R}_i,\mathbf{t}_i)$1 is a robust loss (Hanning et al., 6 Aug 2025).

Two additional geometric terms complement feature alignment. First, an edge term samples 3D points along the 12 cuboid edges, projects them into the images, and evaluates a predicted edge map: $(\mathbf{R}_i,\mathbf{t}_i)$2 Second, a vanishing-point term constrains the three cuboid axis directions through image line segments detected with DeepLSD. The paper writes

$(\mathbf{R}_i,\mathbf{t}_i)$3

with

$(\mathbf{R}_i,\mathbf{t}_i)$4

The full objective is

$(\mathbf{R}_i,\mathbf{t}_i)$5

with hyperparameters such as $(\mathbf{R}_i,\mathbf{t}_i)$6 and $(\mathbf{R}_i,\mathbf{t}_i)$7 reported in the paper (Hanning et al., 6 Aug 2025).

3. Network architecture, optimization, and training

PixCuboid uses a ResNet‑101 encoder with a U-Net style decoder. For each image and each scale, the network predicts a dense feature map $(\mathbf{R}_i,\mathbf{t}_i)$8, a feature confidence map $(\mathbf{R}_i,\mathbf{t}_i)$9, an edge map $\mathcal{C}$0, and an edge confidence map $\mathcal{C}$1 (Hanning et al., 6 Aug 2025).

Inference proceeds in a coarse-to-fine manner. At each of three scales—coarse, medium, and fine—the method solves

$\mathcal{C}$2

using a Levenberg–Marquardt optimizer. Each scale uses 15 LM iterations, and the solution from one scale initializes the next. The paper states that LM damping is learned per cuboid parameter during training (Hanning et al., 6 Aug 2025).

The training signal is defined through 2D–3D correspondences sampled from a ground truth semantic mesh. Points labeled wall, floor, and ceiling are projected into the images, and the cuboid is penalized according to the reprojection error induced by the cuboid warp: $\mathcal{C}$3 The total loss aggregates supervision across scales: $\mathcal{C}$4 If optimization fails at a given scale, gradients to subsequent scales are stopped for that example by setting the corresponding $\mathcal{C}$5 to zero (Hanning et al., 6 Aug 2025).

The paper also describes a pre-training stage for the edge branch. Before end-to-end optimization-in-the-loop training, the predicted edge maps are regressed with a weighted MSE to line drawings of the ground truth cuboid projection. This initialization makes the edge term usable from the start of training. The stated effect of end-to-end training is that the learned feature maps yield large convergence basins and smooth loss landscapes in the alignment, enabling initialization from simple heuristics (Hanning et al., 6 Aug 2025).

4. Initialization and practical inference pipeline

PixCuboid does not require a learned initializer. Its initialization uses camera poses alone. The method computes the mean negative camera $\mathcal{C}$6-axis across views, treats that average direction as the cuboid’s vertical axis, samples two orthogonal basis vectors in the orthogonal plane to form the remaining axes, and initializes face offsets so that all camera centers are inside the cuboid with a margin of 2.5 m. It then runs a few LM iterations using only the vanishing-point term $\mathcal{C}$7 to refine orientation and recomputes the offsets so that cameras remain inside the cuboid (Hanning et al., 6 Aug 2025).

The inference pipeline described in the paper is:

1. Undistort images and resize them to a fixed height, reported as 512 px while retaining aspect ratio.
2. Pass each image through the network to obtain multi-scale feature, confidence, and edge maps.
3. Compute $\mathcal{C}$8 from camera poses with vanishing-point refinement.
4. Run coarse-to-fine LM optimization over the full objective $\mathcal{C}$9.
5. Slightly expand the cuboid after each LM step, if necessary, so that cameras are at least 0.1 m away from faces.
6. Output the final fine-scale cuboid as the room layout (Hanning et al., 6 Aug 2025).

A plausible implication is that the system is structurally closer to learned geometric optimization methods than to direct layout regression. That interpretation is consistent with the paper’s explicit comparison to featuremetric pose-optimization paradigms.

5. Benchmarks, metrics, and empirical performance

PixCuboid introduces two new multi-view cuboid benchmarks. The ScanNet++ benchmark is derived from ScanNet++ v2. Ground truth cuboids are fitted to mesh vertices labeled floor, wall, and ceiling, and only scenes where a cuboid is a reasonable approximation and optimization succeeded are retained. The resulting split contains 391 training scenes, 10 validation scenes, and 18 test scenes. For each scene, the paper generates tuples of 10 random DSLR images; training uses 250 tuples per scene, while validation and test use 20 tuples per scene (Hanning et al., 6 Aug 2025).

The 2D‑3D‑Semantics benchmark is built from spaces in the 2D‑3D‑Semantics dataset. For each space, a cuboid is fitted from point clouds with floor, wall, and ceiling labels, then manually filtered to keep cuboid-like spaces. The benchmark contains 160 spaces and is used only for evaluation. It supports both panorama-based methods and perspective-view methods by splitting each panorama into four perspective images with $\mathcal{C}$0 horizontal field of view (Hanning et al., 6 Aug 2025).

The reported evaluation metrics are 3D IoU, Chamfer distance, rotation error, AUC of rotation accuracy at $\mathcal{C}$1 and $\mathcal{C}$2, pixel-wise depth RMSE, surface-normal accuracy measured as the fraction of pixels with angular error below $\mathcal{C}$3, and runtime (Hanning et al., 6 Aug 2025).

The following table summarizes the main reported results.

Dataset / method	IoU	Chamfer	Depth RMSE
ScanNet++ / Total3D	34.5	1.44 m	0.86 m
ScanNet++ / Implicit3D	30.8	1.60 m	1.02 m
ScanNet++ / PixCuboid	87.2	0.22 m	0.09 m
2D‑3D‑Semantics / Deep3DLayout	58.8	0.68 m	0.44 m
2D‑3D‑Semantics / LED$\mathcal{C}$4-Net	68.7	0.45 m	0.40 m
2D‑3D‑Semantics / PixCuboid	89.0	0.18 m	0.10 m

On ScanNet++ test tuples with 5 input images, PixCuboid reports IoU $\mathcal{C}$5, Chamfer $\mathcal{C}$6 m, rotation error $\mathcal{C}$7, AUC@$\mathcal{C}$8 of $\mathcal{C}$9 and $\mathbf{R} \in SO(3)$0, depth RMSE $\mathbf{R} \in SO(3)$1 m, surface-normal accuracy $\mathbf{R} \in SO(3)$2, and runtime $\mathbf{R} \in SO(3)$3 s. Total3D and Implicit3D are reported at IoU $\mathbf{R} \in SO(3)$4 and $\mathbf{R} \in SO(3)$5, with substantially higher Chamfer and RMSE values (Hanning et al., 6 Aug 2025).

On 2D‑3D‑Semantics, PixCuboid reports IoU $\mathbf{R} \in SO(3)$6, Chamfer $\mathbf{R} \in SO(3)$7 m, rotation error $\mathbf{R} \in SO(3)$8, AUC@$\mathbf{R} \in SO(3)$9 of $\mathbf{d} \in \mathbb{R}^6$0 and $\mathbf{d} \in \mathbb{R}^6$1, depth RMSE $\mathbf{d} \in \mathbb{R}^6$2 m, surface-normal accuracy $\mathbf{d} \in \mathbb{R}^6$3, and runtime $\mathbf{d} \in \mathbb{R}^6$4 s. The panorama-based baselines Deep3DLayout, LED$\mathbf{d} \in \mathbb{R}^6$5-Net, and PSMNet all report lower IoU and higher Chamfer and RMSE under the paper’s evaluation protocol (Hanning et al., 6 Aug 2025).

The ablations identify several operative factors. Edge cost alone already gives good results; adding the featuremetric term improves success rate; adding the vanishing-point term further reduces rotation error and improves normals and IoU. Confidence-guided sampling performs slightly best among the tested sampling strategies. With increasing numbers of views, IoU improves from approximately $\mathbf{d} \in \mathbb{R}^6$6 with 2 views to approximately $\mathbf{d} \in \mathbb{R}^6$7 with 10 views, with strong gains from 2 to 3 views (Hanning et al., 6 Aug 2025).

6. Relation to related cuboid-estimation research

PixCuboid belongs to a broader line of work that uses cuboids as structural abstractions, but its formulation differs from both single-view cuboid fitting and cuboid-first object reconstruction. “Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images” formulates scene abstraction as a robust estimator guided by a neural network, fitting multiple cuboids sequentially to 3D features derived from single RGB images, with an occlusion-aware distance metric and no cuboid annotations during training (Kluger et al., 2021). That method targets cluttered real-world indoor scenes and abstracts them as a set of cuboids, whereas PixCuboid optimizes a single room cuboid from multiple posed views (Hanning et al., 6 Aug 2025).

A different cuboid-first pattern appears in “Parcel3D: Shape Reconstruction from Single RGB Images for Applications in Transportation Logistics,” where CubeRefine R‑CNN predicts a 3D bounding box for parcels and then refines it into a mesh to capture damage and deformation from a single RGB image (Naumann et al., 2023). The shared principle is the use of a cuboid prior, but the object class, supervision, and downstream task differ: Parcel3D focuses on parcel damage and tampering detection, while PixCuboid targets room layout estimation (Hanning et al., 6 Aug 2025).

This suggests a useful distinction within cuboid-based vision systems. In the single-view literature represented in the provided sources, cuboids are used either as robust scene primitives or as object priors; in PixCuboid, the cuboid is the explicit room-layout variable optimized by multi-view feature alignment. That placement at the intersection of room layout estimation, featuremetric alignment, and optimization-in-the-loop is presented by the paper as its principal novelty (Hanning et al., 6 Aug 2025).

7. Extensions, limitations, and interpretation

The paper reports a straightforward multi-room extension despite training only on single cuboids. In a multi-room Replica scene with four rooms, PixCuboid processes a long monocular trajectory with known poses, maintains a set of existing cuboids, optimizes a new cuboid together with the existing ones over a sliding window of 8 frames, and constrains shared orientation, shared floor and ceiling heights, and shared wall positions where appropriate. Candidate cuboids with IoU greater than $\mathbf{d} \in \mathbb{R}^6$8 against existing cuboids are rejected as duplicates. The resulting set of aligned cuboids approximates a multi-room apartment or office (Hanning et al., 6 Aug 2025).

The stated strengths are multi-view metric reconstruction, end-to-end learned features tailored for alignment, high accuracy on the proposed benchmarks, flexibility with arbitrary numbers of views, and strong cross-dataset generalization from ScanNet++ to 2D‑3D‑Semantics. The stated limitations are the cuboid assumption, the requirement for known camera poses, dependence on multiple views, reliance on Manhattan structure for the vanishing-point term, and the optimization overhead of LM relative to pure feed-forward regressors (Hanning et al., 6 Aug 2025).

The paper’s future directions include extending the parameterization beyond cuboids to general Manhattan layouts or non-Manhattan structures, jointly optimizing camera poses and layout, integrating with dense reconstruction, and transferring the featuremetric cuboid-fitting paradigm to larger-scale or outdoor environments (Hanning et al., 6 Aug 2025).

Taken together, PixCuboid defines a specific form of room-layout estimation: a cuboid-valued geometric optimization problem over multi-view deep features. Its central claim is not merely that dense learned features can support layout estimation, but that features can be trained specifically so that cuboid alignment becomes stable under simple initialization heuristics. Within the scope described by the paper, this makes PixCuboid a representative example of optimization-based, multi-view, cuboid-parameterized indoor geometry estimation (Hanning et al., 6 Aug 2025).