Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments

Abstract: LLMs have demonstrated remarkable capabilities in 3D indoor synthesis for Manhattan environments. However, existing methods often fail to capture plausible object layout patterns in non-Manhattan settings, primarily because they struggle to model non-orthogonal spatial relationships, leading to high geometric violations and low physical fidelity. To address this challenge, we propose SPG-Layout, a novel text-driven framework designed to generate physically plausible indoor scenes within complex non-Manhattan environments. Specifically, we first utilize statistical priors of object distributions to guide the training process, enhancing environmental understanding and fidelity. Furthermore, mirroring human design workflows, we adopt a hierarchical layout strategy that prioritizes the placement of large objects, thereby substantially minimizing layout violations. By synergizing these components, SPG-Layout achieves a balanced optimization of semantic realism and physical plausibility. To evaluate performance in these complex settings, we constructed a new benchmark comprising 500 diverse non-Manhattan environments. Extensive experiments demonstrate that SPG-Layout consistently and significantly outperforms existing methods across both Manhattan and non-Manhattan environments. The code will be publicly released.

• The paper introduces SPG-Layout, which uses reinforcement learning with spatial prior guidance and hierarchical layout strategies to substantially reduce geometric violations in non-Manhattan scene synthesis.
• It employs a two-stage training approach that combines supervised fine-tuning of structured scene representations with RL optimization, achieving an 86–96% reduction in layout violation metrics.
• The work establishes a novel 500-scene non-Manhattan benchmark, paving the way for advanced research in text-driven indoor scene synthesis and practical applications in AR/VR, robotics, and design.

Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments: An Expert Analysis

Introduction and Motivation

Text-driven 3D indoor scene synthesis has seen rapid progress, primarily leveraging the semantic compositionality of LLMs to convert natural language instructions into scene layouts. Most contemporary frameworks assume axis-aligned, grid-like Manhattan world geometries, which vastly simplifies boundary constraints but fails to generalize to real architecture wherein non-Manhattan—arbitrarily angled or curved—environments are prevalent.

Figure 1: Contrasting Manhattan (orthogonal, grid-like) with non-Manhattan (irregular, curved, oblique) environments, highlighting the increased geometric reasoning demands of the latter.

Existing LLM-based and diffusion-based layout syntheses, including LayoutGPT, LayoutVLM, InstructScene, and recent diffusion models, typically regress object coordinates based on implicit Manhattan priors, resulting in high rates of geometric violations and physically implausible layouts when applied to non-Manhattan domains. Crucially, the inability to model complex boundary constraints and non-axis-aligned object-object relationships leads to both spatial fragmentation and functional incoherence.

SPG-Layout: Framework Overview

The paper introduces SPG-Layout, a new scene synthesis paradigm explicitly targeting text-to-3D generation within non-Manhattan environments. The architecture comprises three core innovations:

1. Spatial Prior Guidance (SPG): Encodes statistical object-boundary and object-object relationship priors into the reward function, providing dense, interpretable feedback for reinforcement learning (RL)-based scene generation.
2. Hierarchical Layout Strategy (HLS): Inspired by expert interior design procedures, object placements are staged from largest to smallest. This minimizes spatial fragmentation and guarantees that space-consuming objects anchor the spatial skeleton, dramatically reducing collision and infeasibility rates.
3. Two-Stage Training (Supervised Fine-tuning + RL): An LLM is first fine-tuned to emit structured scene representations (SSR), then post-trained using Group Relative Policy Optimization (GRPO) driven by composite geometric, semantic, and format conformance rewards.

Figure 2: SPG-Layout’s architecture, from structured scene specification to RL-optimized scene evolution and hierarchical object placement.

Notably, the paper also curates a new 500-scene benchmark of real, human-annotated non-Manhattan layouts, offering a crucial resource for future benchmarking.

Structured Scene Representation and Problem Formulation

Scene generation is formulated as conditional sequence modeling, where the SSR explicitly disentangles the room boundary mesh/sequences from sets of object tuples parameterized by description, scale, position, and rotation. The core generation problem reduces to, at each timestep, producing an SSR-compliant increment maximizing the conditional likelihood under both prompt constraints and geometric requirements.

Spatial Prior Guidance: Statistical Reward Modeling

LLMs alone lack sufficient geometric inductive bias for physically plausible, human-preferred placements, especially in complex, non-rectilinear domains. SPG remedies this by supplying explicit reward terms:

• $\mathcal{B}(\cdot)$: Validates object positions against objective, category-conditioned boundary proximity distributions.
• $\mathcal{A}(\cdot)$: Captures semantic-aware pairwise affinities, especially for canonical object pairs (e.g., bed–nightstand), via a Gaussian-model reward on inter-object distances.
• An attention mechanism adaptively weights prior influences from previously placed objects, outperforming naive mean/max aggregation baselines.

These reward signals are not hard constraints but serve as strong statistical regularizers, bridging the gap between LLM generative semantics and geometric physicality.

Figure 3: SPG fuses geometric boundary priors and co-occurrence affinities into computable RL reward signals, improving spatial coherence.

Hierarchical Layout Strategy

HLS addresses the critical issue of incremental infeasibility: uncoordinated early placements of small objects can block feasible placements of future large objects. By spatially sorting objects and strictly enforcing a large-to-small placement sequence, HLS operationalizes an effective constraint propagation scheme, further enabling robust single-object and full-scene synthesis under highly irregular geometry.

Figure 4: Illustration of the hierarchical placement protocol: objects are ordered by area and allocated coarse-to-fine.

Non-Manhattan Dataset Construction

Given the scarcity of suitable non-Manhattan benchmarks, the paper details a semi-automated pipeline combining large-scale floorplan mining, coarse scene bootstrapping with adapted Manhattan-trained models, and meticulous manual refinement for physical and semantic plausibility. The compiled dataset (500 scenes) enables, for the first time, rigorous evaluation on non-rectilinear domains.

Figure 5: Pipeline for non-Manhattan scene creation, blending automatic bootstrapping and human refinement.

Experimental Evidence and Quantitative Evaluation

Extensive experiments demonstrate SPG-Layout achieves:

• 86–96% reduction in layout violation metrics (OOB, MBL, VBL) compared to prior state-of-the-art, both on single-object addition and full scene synthesis tasks, in non-Manhattan settings.
• Marginal or no loss of SOTA performance on conventional Manhattan scenes, establishing strong generalization.
• Improved object-architecture (OAR) and object-object relationship (OOR) scores, directly correlating with human design intent and spatial usability.
• Favorable human preference in a 20-subject user study, with SPG-Layout overwhelmingly chosen for realism and spatial rationality.

Ablation studies confirm that SPG (especially with attention-based aggregation) and HLS contribute the vast majority of performance gain. The removal of SPG or HLS results in substantial degradation of both layout fidelity and violation metrics.

Figure 6: Qualitative comparisons—SPG-Layout produces collision-free, semantically and spatially plausible layouts in challenging non-Manhattan settings, unlike existing baselines.

Figure 7: End-to-end scene visualizations on the new benchmark, with layouts conditioned on complex, real-world text specifications.

Limitations and Failure Modes

SPG-Layout still struggles with:

• Long-tail categories not sufficiently represented in prior estimation.
• Fine-grained orientation and arrangement of small, asymmetric objects—functional errors remain, even in collision-free scenes.
• Non-local rearrangements in single-object edits, due to global HLS re-allocations.
• Computational overhead from voxel-based geometric rewards during RL.

Figure 8: Example failure—fragmented, cluttered arrangement despite absence of collisions, underscoring aesthetic and functional challenges in dense scenes.

Practical and Theoretical Implications

The work demonstrates that integrating structured statistical priors and hierarchical procedural constraints into LLM-based text-to-scene pipelines is critical for scaling up to "in-the-wild" architectures. The explicit SPG reward represents a tractable path toward integrating human spatial knowledge into generative models, while HLS offers a principled solution to the recursive feasibility issues endemic to autoregressive scene construction.

Practically, this advances applications in autonomous robotics, interior design, AR/VR content creation, and architectural prototyping, especially for complex, custom or historical layouts. The modularity of SSR, SPG, and HLS suggests extensibility to other modalities (e.g., outdoor scenes, industrial environments) and multi-agent scene negotiation.

Theoretically, this establishes a baseline for reward shaping in text-conditioned structural synthesis, as well as confirming the utility of RL post-training for physical and semantic alignment.

Figure 9: SPG-Layout’s SSR enables seamless editing, removal, and replacement workflows—beyond forward synthesis.

Conclusion

SPG-Layout introduces a scalable, RL-driven framework for text-conditioned 3D indoor scene generation that, for the first time, robustly addresses the unique challenges of non-Manhattan environments. By fusing explicit statistical priors, hierarchical generation strategies, and two-stage training, it delivers state-of-the-art spatial, semantic, and physical plausibility without sacrificing generalization to conventional scenarios. This work constitutes a rigorous foundation for subsequent research on structure-aware, instruction-driven scene synthesis in unconstrained domains.

For precise architectural details, reward function formalism, dataset provenance, and comprehensive benchmarking, readers are referred directly to "Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environments" (2607.02407).