SemLayoutDiff: Unified 3D Scene Diffusion

• SemLayoutDiff is a diffusion-based model that generates 3D indoor scenes using a top-down semantic map paired with explicit per-object attributes.
• The framework first diffuses semantic maps with architectural masks and then uses cross-attention to predict furniture placement, ensuring practical and unobstructed layouts.
• Evaluation on the 3D-FRONT dataset shows significant improvements in physical plausibility and distribution matching metrics compared to baseline methods.

SemLayoutDiff is a unified diffusion-based model for synthesizing diverse 3D indoor scenes across multiple room types. It represents a scene through a top-down semantic map together with per-object attributes, and it conditions scene synthesis explicitly on room masks, including architectural elements such as doors and windows. The pipeline first generates a coherent semantic map with a categorical diffusion model, then uses a cross-attention-based network to predict furniture placements that respect the synthesized layout, so that resulting arrangements remain practical and unobstructed (Sun et al., 26 Aug 2025).

1. Representational basis

SemLayoutDiff formulates indoor scene synthesis around two coupled representations: a semantic map $\mathcal{S}$ and an instance-attribute set $\mathcal{I}=\{O_i\}$. The semantic map is a 2D top-down segmentation map, $\mathcal{S}\in\mathbb{R}^{H\times W}$, in which each pixel corresponds to a fixed real-world space of $0.01$m per pixel. Each pixel stores a semantic category, covering both furniture categories and architectural elements. Beyond object categories $C$, the representation includes special categories for floor, door, window, and void, giving $K=C+4$ total categories; in the reported dataset configuration, $K=38$ (Sun et al., 26 Aug 2025).

Each object instance $O_i$ carries a category $c_i$, a size $s_i\in\mathbb{R}^3$, a position $\mathcal{I}=\{O_i\}$0, and an orientation $\mathcal{I}=\{O_i\}$1. Orientation is discretized to four major axes: $\mathcal{I}=\{O_i\}$2, $\mathcal{I}=\{O_i\}$3, $\mathcal{I}=\{O_i\}$4, and $\mathcal{I}=\{O_i\}$5. This coupling of a dense semantic layout with explicit object attributes is central to the model’s scope: the semantic map captures room-scale organization, while the instance attributes support 3D placement and asset retrieval (Sun et al., 26 Aug 2025).

A defining feature of the representation is that architectural elements are part of the semantic map itself rather than external constraints. This allows room shape, doors, and windows to participate directly in generation and conditioning. In the reported formulation, that design is the key mechanism by which SemLayoutDiff supports architectural compliance during scene synthesis (Sun et al., 26 Aug 2025).

2. Categorical diffusion over semantic maps

The first stage is a semantic layout generator based on a multinomial diffusion model operating on categorical data. Rather than diffusing continuous coordinates or latent vectors, the model diffuses semantic labels over the top-down map. The forward process is written as

$\mathcal{I}=\{O_i\}$6

where $\mathcal{I}=\{O_i\}$7 denotes the categorical distribution and $\mathcal{I}=\{O_i\}$8 defines the diffusion schedule. At each forward step, a portion of elements is randomly replaced with other category labels according to the schedule $\mathcal{I}=\{O_i\}$9; inference reverses this process from random category noise back to a valid semantic map (Sun et al., 26 Aug 2025).

The denoiser is a U-Net with explicit conditioning on both room masks and room type. The room mask $\mathcal{S}\in\mathbb{R}^{H\times W}$0 can be categorical, with labels for void, floor, door, and window; binary, with floor and void only; or absent in unconditional generation. A room-type embedding $\mathcal{S}\in\mathbb{R}^{H\times W}$1 specifies the room class, such as bedroom, dining room, or living room. Conditioning is realized by adding the architectural and room-type embeddings to the noise or timestep embeddings in the U-Net. The training objective is the KL divergence between the true and predicted reverse categorical transitions:

$\mathcal{S}\in\mathbb{R}^{H\times W}$2

This stage supports three generation regimes. In unconditional generation, the model synthesizes both architecture and objects from noise. In the floormask setting, only the floor boundary is given and the model fills in the room accordingly. In the archmask setting, the full architectural boundary including doors and windows is provided, and the model generates layouts that comply with those structures (Sun et al., 26 Aug 2025).

3. Cross-attention attribute prediction and scene assembly

After sampling the semantic map, SemLayoutDiff performs instance extraction before predicting 3D object attributes. The semantic layout is processed with connected-component analysis on a per-category basis, and category-specific size thresholds are used to filter noise. The resulting instance masks, together with the sampled semantic map, form the input to the Attribute Prediction Model (APM) (Sun et al., 26 Aug 2025).

The APM contains parallel encoders for global and local information. An encoder $\mathcal{S}\in\mathbb{R}^{H\times W}$3 maps the semantic layout to a feature representation $\mathcal{S}\in\mathbb{R}^{H\times W}$4, while an encoder $\mathcal{S}\in\mathbb{R}^{H\times W}$5 maps an instance mask to $\mathcal{S}\in\mathbb{R}^{H\times W}$6. Cross-attention then combines them, with $\mathcal{S}\in\mathbb{R}^{H\times W}$7 as query and $\mathcal{S}\in\mathbb{R}^{H\times W}$8 as key and value, yielding an instance-level feature vector for each object. Prediction heads implemented as MLPs regress size and position and classify orientation. The training loss is

$\mathcal{S}\in\mathbb{R}^{H\times W}$9

where $0.01$0 and $0.01$1 are MSE losses for size and position, and $0.01$2 is a cross-entropy loss for orientation (Sun et al., 26 Aug 2025).

The predicted attributes include vertical size, vertical position, and orientation class. These predictions are then used for 3D asset retrieval: for a predicted object, the system selects a real 3D model from a category-specific asset set by minimizing size mismatch,

$0.01$3

The summary describes retrieval from a database such as 3D-FUTURE. In effect, the second stage translates a 2D semantic layout into a furnished 3D scene by combining instance geometry, room context, and category-constrained asset selection (Sun et al., 26 Aug 2025).

4. Architectural conditioning and unified generation

A central claim of SemLayoutDiff is that prior approaches either require separate models for each room type or cannot generate layouts in accordance with architectural constraints. SemLayoutDiff addresses both limitations with a single architecture that can operate across multiple room types and multiple conditioning regimes. The same model can be trained for all room types and all conditioning modes, including a mixed condition model, rather than maintaining separate generators for bedrooms, living rooms, dining rooms, or different mask settings (Sun et al., 26 Aug 2025).

Architectural conditioning is not merely a boundary prior. Because doors and windows are encoded as semantic categories and injected as conditioning at every diffusion step, the generated layouts are constrained both spatially and functionally. The reported description states that this ensures objects are never placed out of bounds and, crucially, not blocking doors or windows. The model therefore learns architectural constraints directly from data instead of enforcing them only at post-processing time (Sun et al., 26 Aug 2025).

This architectural formulation also clarifies the model’s practical scope. SemLayoutDiff is not just a category map generator, since it continues from the semantic map to object-level 3D attribute prediction and asset placement. Conversely, it is not a purely object-set model, because the semantic map explicitly mediates room-scale structure before furniture placement. That two-stage decomposition is the mechanism by which the framework combines room-mask conditioning, semantic coherence, and practical furniture arrangement (Sun et al., 26 Aug 2025).

5. Evaluation on 3D-FRONT

Experiments are reported on the 3D-FRONT dataset using ATISS’s split, with 4,616 rooms spanning bedroom, living room, and dining room categories and 38 semantic classes. The baselines are DiffuScene and MiDiffusion, both adapted when possible for unified multi-room and mask-conditional settings. Evaluation covers distribution matching through FID, KID, SCA, and CKL, and physical plausibility through out-of-bounds ratios $0.01$4, collision rate (COL), and navigability (NAV) (Sun et al., 26 Aug 2025).

Under architecture-mask conditioning, SemLayoutDiff reports markedly improved physical plausibility. DiffuScene yields $0.01$5, $0.01$6, $0.01$7, and $0.01$8; MiDiffusion reports $0.01$9, $C$0, $C$1, and $C$2; SemLayoutDiff reports $C$3, $C$4, $C$5, and $C$6, respectively. On distribution matching under the same conditioning, DiffuScene reports $C$7, $C$8, $C$9; MiDiffusion reports $K=C+4$0, $K=C+4$1, $K=C+4$2; and SemLayoutDiff reports $K=C+4$3, $K=C+4$4, and $K=C+4$5 (Sun et al., 26 Aug 2025).

Qualitative results are described as showing unobstructed passageways at doors and windows, coherent furniture clusters, and accurate room-object relationships. The unconditional mode generates varied plausible room geometries, while conditioned generation produces variations fitted to a given plan. In a user study on 40 samples, SemLayoutDiff was rated best 80% of the time, compared with 11% for DiffuScene and 8% for MiDiffusion. Fine-grained annotation also found fewer architecture violations, blocked doors, and implausible layouts. Training and inference are described as competitive with, or better than, the baselines, while the unified model remains more efficient than training separate models per room or condition (Sun et al., 26 Aug 2025).

6. Position within diffusion-based layout research

SemLayoutDiff sits within a broader diffusion-based literature on layout generation, layout editing, and structured scene synthesis, but its problem definition is more specific than many neighboring approaches. In 2D layout generation, LDGM frames arbitrary missing or coarse element attributes as intermediate diffusion states and uses a decoupled diffusion process over category, position, and size to unify conditional and unconditional generation in a single transformer-based model (Hui et al., 2023). Dolfin instead performs diffusion directly in the original input space with a Transformer backbone and introduces both a bi-directional model and an autoregressive variant, Dolfin-AR, to capture alignment, size, and overlap correlations in graphic layouts and other geometric structures such as line segments (Wang et al., 2023). For document design, diffusion has also been applied in the sequence domain rather than the image domain, together with the Doc-EMD metric for comparing document layouts across heterogeneous categories (He et al., 2023).

A separate line of work addresses editing rather than synthesis. “Continuous Layout Editing of Single Images with Diffusion Models” introduces masked textual inversion and a training-free optimization method to rearrange objects in an existing image while preserving their visual properties (Zhang et al., 2023). “Consistent Image Layout Editing with Diffusion Models” uses a multi-concept learning scheme, semantic consistency within intermediate diffusion features, and a layout-friendly initialization noise to improve appearance consistency under layout rearrangement of real images (Xia et al., 9 Mar 2025). These systems target existing images; SemLayoutDiff targets 3D indoor scene synthesis from semantic layouts and room masks (Sun et al., 26 Aug 2025).

There are also scene-planning and 3D-editing systems that employ different intermediate representations. LayoutAgent combines vision-language preprocessing, scene graph construction, and compositional diffusion to synthesize object bounding boxes for spatial layouts before foreground-conditioned image generation (Fan et al., 24 Sep 2025). EditRoom represents a 3D room as a semantic graph plus a layout matrix and uses LLM-parameterized graph diffusion to execute rotate, translate, scale, replace, add, and remove edits in response to natural-language commands (Zheng et al., 2024). DiffX extends layout guidance to cross-modal generation in a modality-shared latent space, supporting layout-guided RGB+X synthesis with a Joint-Modality Embedder for text-layout interaction (Wang et al., 2024). Against this background, SemLayoutDiff is distinguished by a two-stage pipeline that first generates a categorical top-down semantic layout under architectural conditioning and then predicts furniture attributes for 3D placement, yielding a unified model for multi-room indoor scene synthesis (Sun et al., 26 Aug 2025).