Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation

Abstract: Building structured 3D scene layouts from a single image requires reconciling visual observations with physical and spatial constraints, a challenge that is difficult to address with direct prediction alone. In this work, we formulate monocular 3D layout estimation as a perceive-then-plan problem with vision-LLMs, where a Perceiver first grounds the 3D objects and then a Planner iteratively refines the scene hypothesis through actions that improve physical plausibility while preserving consistency with the input image. We propose Layout-as-Policy (LaP), which casts the planning stage as a policy learning problem: 3D layouts are represented as structured states, and refined via discrete actions such as translation, rotation, and rescaling. Starting from an observation-aligned initialization with the geometry-enhanced Perceiver, the LaP Planner is trained to produce action sequences that progressively resolve geometric inconsistencies and enforce realistic spatial relations. To enable effective learning, we combine supervised trajectory initialization with preference-based optimization, allowing the model to learn corrective behaviors without requiring explicit reward engineering. This formulation transforms layout estimation from a one-shot prediction task into an iterative refinement process, enabling better handling of global constraints and complex object interactions. Experiments demonstrate that our approach produces layouts that are more physically coherent and better aligned with visual observations, while naturally supporting downstream tasks such as scene editing and manipulation.

• The paper introduces a two-stage Layout-as-Policy framework that combines geometry-enhanced 3D object grounding with iterative, action-based layout correction instead of one-shot prediction.
• The 8B system improves reprojection IoU from 0.54 to 0.67, reduces scene collisions from 3.62 to 1.44, and lowers rotation error from 17.74° to 8.62° through multi-round refinement.
• The method shows that grid canonicalization, supervised warm-up, and direct preference optimization help produce concise corrective actions while supporting physically plausible indoor scene reconstruction and language-guided editing.

Overview

This paper reformulates monocular 3D scene layout estimation as a two-stage perceive-then-plan process built on vision-LLMs (VLMs). Rather than regressing a complete 3D layout in one forward pass, the framework first grounds detected objects in 3D with a geometry-enhanced Perceiver, then iteratively refines the resulting layout through discrete corrective actions selected by a learned policy, termed Layout-as-Policy (LaP). The central claim is that layout estimation is better handled as sequential decision-making over structured scene states than as direct prediction, because global constraints such as non-collision and valid support are interdependent and difficult to enforce in a single shot.

Method

The pipeline takes an image and 2D detections and produces a structured 3D layout. In the perceive stage, the Perceiver lifts each 2D detection into a 3D box using a local-axis coordinate representation (bottom-face axes $ax_x$, $ax_z$ with $ax_y = ax_x \times ax_z$), which avoids requiring the VLM to regress raw box vertices. A lightweight geometry-aware modulation module injects features from a pretrained geometry encoder (VGGT) into the VLM's visual features via FiLM-style scaling and a spatial gate, keeping the fused features close to the original visual features at initialization. Grounded boxes are then canonicalized into a gravity-aligned frame—gravity estimated via SVD over bottom-face normals—and discretized into a grid representation (0.1 m cells, 24 yaw bins), yielding an initial structured layout $\hat{\mathcal{L}}_{3d}$.

In the plan stage, the LaP Planner treats the layout as a state and applies actions from a compact vocabulary: SELECT($i$), MOVE($dx, dy, dz$), RESIZE($ds$), ROTATE_Y($d\theta$), and STOP. Training proceeds in three steps. First, perturbed layouts are synthesized by sampling random action sequences from ground truth and applying their inverses, providing paired inputs and corrective trajectories. Second, supervised fine-tuning (SFT) on these trajectories gives the Planner basic action-layout-image comprehension. Third, direct preference optimization (DPO) is applied using offline trajectory pairs: ground-truth corrective sequences as preferred samples, and Perceiver-initialized or degraded variants (numerical shifts, nuisance actions, missing actions, wrong action types) as rejected samples. Refinement runs for multiple rounds until STOP or a round limit, letting the model resolve subsets of violations per iteration. A final post-processing step generates a contact scene graph, groups objects into relational bundles, and runs single-pass gravity simulation to enable realistic scene assembly with retrieved or generated assets.

Experimental results

Training uses Hypersim and SUN RGB-D (42,160 filtered training samples; evaluation on 1,000 SUN RGB-D test scenes). Both components are built on Qwen3-VL at 2B and 8B scales.

Perceiver grounding. The Perceiver substantially outperforms both open-source and closed-source baselines. The 2B model achieves Reprojection IoU of 0.54, Precision@IoU=0.25 of 0.91, Precision@IoU=0.5 of 0.62, and average depth error of 0.09 m; the 8B model reaches 0.58 IoU and 0.08 m depth error. By comparison, GPT-4o attains only 0.27 IoU and 0.55 m depth error under direct 3D grounding, and the strongest fine-tuned baseline (Qwen3-VL 8B with SFT) reaches 0.44 IoU and 0.18 m depth error. Ablations show that removing geometry modulation causes the largest degradation (IoU drops to 0.36, depth error rises to 0.18), while removing the local-axis representation drops IoU to 0.41—both components are necessary for accurate monocular lifting.

LaP refinement. Starting from Perceiver-initialized layouts (Reproj. IoU 0.54, SVR 11.63%, 3.62 collisions, rotation error 17.74°), converged LaP refinement improves all metrics: the 8B Planner reaches 0.67 IoU (+0.13), SVR 4.12% (−7.52 points), 1.44 collisions (−2.18), and rotation error 8.62° (−9.12°), with depth error improving to 0.07 m. Notably, a trained one-shot VLM direct-refinement baseline with identical training data yields almost no benefit (+0.01 IoU) and increases depth error to 0.11 m; multi-round direct prediction degrades further (IoU 0.56 after 3 rounds). Rule-based refinement eliminates collisions and support violations entirely but destroys visual alignment (IoU falls to 0.26, depth error 0.24 m), since it reasons without image conditioning. GPT-4o iterative refinement reduces IoU below initialization. These comparisons support the paper's claim that iterative action-based refinement, not direct prediction, is what enables joint physical plausibility and visual alignment.

Ablations on the Planner. Removing grid-based canonicalization costs 0.07 IoU and nearly doubles rotation error; removing SFT warm-up costs 0.09 IoU; removing DPO leaves IoU at 0.60 but raises SVR back to 8.77% and increases average action count from 3.67 to 4.73, indicating DPO's main contribution is learning minimal, sufficient corrective sequences rather than raw capability.

Downstream use. The shared action vocabulary supports language-guided scene editing without architectural change, and assembled scenes compare favorably against layouts recovered by SAM 3D using the same assets.

Limitations and open questions

The authors identify several constraints. The pipeline depends on off-the-shelf 2D detectors as input, so detection errors and missed objects propagate through both stages; tighter coupling between detection and 3D reasoning remains unaddressed. Evaluation is restricted to indoor scenes with well-defined support hierarchies, where gravity-aligned canonicalization and contact graphs are informative—generalization to less structured environments is left open. The Perceiver and Planner are trained separately, and whether joint optimization would improve spatial reasoning is an open question. Additionally, the Planner comparison lacks external baselines specifically designed for VLM-based layout refinement, so conclusions rest on internally constructed alternatives (one-shot direct, rule-based, GPT-4o iterative). Extension to multi-view or video inputs, which could resolve depth ambiguities that remain under-constrained monocularly, is also deferred.

Conclusion

This paper recasts monocular 3D layout estimation as policy learning over discretized scene states, combining a geometry-enhanced Perceiver for observation-aligned initialization with a DPO-trained LaP Planner for iterative physical correction. The reported gains—particularly the contrast between convergent action-based refinement and the failure of one-shot direct prediction under matched data—are consistent with the paper's premise that global geometric constraints favor sequential decision-making. The action-level interface additionally provides a reusable mechanism for editing and manipulation, though dependence on upstream detections and indoor-specific assumptions bound the current scope of the results.