GraLa3D: Layout-Guided 3D Scene Generation

• GraLa3D is a framework for generating complex 3D scenes from text using scene graphs, explicit 3D layouts, and diffusion-guided 3D optimization.
• It decomposes scenes into single-object nodes and interactive super-nodes, enabling precise control of spatial arrangements and object interactions.
• The three-stage pipeline—scene graph composition, node-to-3D generation, and mesh harmonization—ensures both geometric consistency and global texture coherence.

Searching arXiv for the GraLa3D paper and closely related papers for accurate citation support. GraLa3D, short for Scene Graph and Layout Guided 3D Scene Generation, is a framework for generating complex 3D scenes from a text prompt by coupling a scene graph representation, explicit 3D layout bounding boxes, and diffusion-guided 3D optimization (Huang et al., 2024). It is motivated by two limitations identified for complex scene synthesis: the intricate relations between objects and the fact that existing methods are largely based on score distillation sampling (SDS), which constrains the ability to manipulate multiobjects with specific interactions. Given a prompt $y^g$, GraLa3D uses an LLM to parse the scene into object nodes, pairwise relations, and axis-aligned 3D bounding boxes; partitions the graph into single-object nodes and composite super-nodes; optimizes 3D Gaussian Splatting (3DGS) representations under layout, interaction, and localization constraints; and finally harmonizes the assembled scene by mesh conversion and UV-texture refinement under the global prompt.

1. Problem setting and architectural overview

GraLa3D is presented as a three-stage pipeline for text-conditioned complex 3D scene generation (Huang et al., 2024). The input is a prompt $y^g$ describing a complex 3D scene, and the output is a 3D mesh scene. The pipeline consists of Scene Graph Composition, Node-to-3D Generation, and 3D Scene Harmonization.

In Scene Graph Composition, an LLM such as GPT-4 parses $y^g$ into a scene graph

$G=(V,E),$

where $V=\{v_1,\dots,v_n\}$ are object nodes and $E=\{e_{i,j}\}$ are pairwise relations. Simultaneously, the LLM predicts an axis-aligned 3D bounding box

$b_i \subset \mathbb{R}^3,\quad i=1\ldots n,$

for each node, collected in $B=\{b_i\}$. The node set is then decomposed into single-object nodes and super-nodes.

In Node-to-3D Generation, each node is realized as one or more 3DGS models $\theta$, each confined to its bounding box. A 2D text-to-image diffusion prior $\epsilon_\phi$, instantiated in the reported system with MVDream, guides 3D optimization via an Interval Score Matching (ISM) variant of SDS. Additional losses are used so that each object stays within its box, interacting objects exhibit the correct mutual poses, and Gaussian leakage between objects is suppressed by a cross-attention-based localization loss.

In 3D Scene Harmonization, all 3DGS models are converted to meshes via marching cubes, and their UV textures $y^g$0 are jointly refined under the global prompt $y^g$1 using a depth-conditioned ControlNet. This stage is intended to yield coherent final scene textures.

A central design choice is the use of both single-object nodes and composite super-nodes. The paper identifies the modeling of interactions between objects in a super-node, while alleviating appearance leakage across objects within such nodes, as a major contribution. This suggests that GraLa3D should be understood not merely as a layout-constrained generator, but as a framework that explicitly treats object interaction as a first-class optimization target.

2. Scene graph representation and decomposition

The scene graph representation in GraLa3D is defined formally over nodes, edges, and per-object layout data (Huang et al., 2024). Each node $y^g$2 carries a text label $y^g$3 and a bounding box

$y^g$4

Edges $y^g$5 have types “spatial” or “interaction.” Bounding-box layouts are stored as per-object 8-corner lists, while relations are stored as adjacency lists or edge-triples.

The graph is decomposed into two subsets: $y^g$6 Here, $y^g$7 contains single-object nodes connected only by spatial edges, while $y^g$8 contains super-nodes, each defined as a pair of objects with an interaction edge.

This decomposition determines how optimization is organized downstream. Single-object nodes are optimized independently under their own prompts and boxes, whereas super-nodes are optimized jointly so that both the composite interaction and the identity of the individual objects are preserved. A plausible implication is that the graph decomposition functions as an intermediate control structure between natural-language scene specification and geometric optimization, allowing GraLa3D to separate ordinary spatial arrangement from explicitly interactive object configurations.

3. Node-to-3D optimization objectives

The optimization core of GraLa3D is written in terms of 3DGS parameters $y^g$9, a sampled camera $y^g$0, and diffusion timestep $y^g$1 (Huang et al., 2024). Under ISM/SDS guidance, the framework uses

$y^g$2

To impose layout constraints, the 2D projection operator is defined as

$y^g$3

For a single-object node $y^g$4 with 3DGS $y^g$5, prompt $y^g$6, and box $y^g$7, the layout loss is

$y^g$8

where $y^g$9 is the rendered alpha map. The masked ISM term is

$G=(V,E),$0

The resulting single-node objective is

$G=(V,E),$1

For a super-node $G=(V,E),$2 with joint prompt $G=(V,E),$3 and boxes $G=(V,E),$4, the interaction branch is defined over the union of the two 3DGS models: $G=(V,E),$5 For each object branch $G=(V,E),$6, the localization loss is

$G=(V,E),$7

where $G=(V,E),$8 is the cross-attention map for token $G=(V,E),$9, extracted via DAAM. The per-object loss is

$V=\{v_1,\dots,v_n\}$0

The combined super-node objective is

$V=\{v_1,\dots,v_n\}$1

The organization of these losses encodes three distinct constraints: box fidelity, diffusion prior alignment, and intra-super-node disentanglement. This suggests that GraLa3D treats interaction fidelity and object separation as coupled but non-identical objectives.

4. Interaction modeling and appearance leakage mitigation

A key technical issue in super-node optimization is appearance leakage, described as the phenomenon in which parts of one object can “leak” onto the other during joint optimization of $V=\{v_1,\dots,v_n\}$2 (Huang et al., 2024). GraLa3D addresses this by measuring the cross-attention map $V=\{v_1,\dots,v_n\}$3 for each token and enforcing

$V=\{v_1,\dots,v_n\}$4

The rendered mask $V=\{v_1,\dots,v_n\}$5 is therefore aligned with the region attended by its own text token in the diffusion prior.

The paper characterizes this cross-attention localization as a soft regularizer preventing entanglement. In the reported ablations, removing $V=\{v_1,\dots,v_n\}$6 leads to Gaussian entanglement. Removing masked ISM produces incomplete objects truncated by box. These ablations clarify that the framework’s interaction modeling does not rely exclusively on union-level optimization; rather, it requires simultaneous control of the composite prompt and token-specific localization.

A common misconception would be to treat super-nodes as simple grouped objects. The formulation instead indicates that a super-node is an interaction-bearing composite with both a joint branch and per-object branches. A plausible implication is that the super-node mechanism is intended to preserve pose-level relational structure without collapsing token-level object identity.

5. End-to-end workflow and harmonization

The algorithmic summary of GraLa3D begins with the prompt $V=\{v_1,\dots,v_n\}$7, composes a scene graph $V=\{v_1,\dots,v_n\}$8 together with boxes $V=\{v_1,\dots,v_n\}$9, decomposes $E=\{e_{i,j}\}$0 into $E=\{e_{i,j}\}$1 and $E=\{e_{i,j}\}$2, optimizes 3DGS models for each node type, and outputs a 3D mesh scene (Huang et al., 2024). The optimization loop is described procedurally: initialize $E=\{e_{i,j}\}$3 inside its bounding box $E=\{e_{i,j}\}$4; sample camera $E=\{e_{i,j}\}$5 and timestep $E=\{e_{i,j}\}$6; render $E=\{e_{i,j}\}$7 and $E=\{e_{i,j}\}$8; compute the binary mask $E=\{e_{i,j}\}$9; form masked ISM and layout losses; and, for super-nodes, compute union-level interaction loss together with per-part localization and object losses.

After node-level optimization, all 3DGS models are converted to meshes via marching cubes. UV maps $b_i \subset \mathbb{R}^3,\quad i=1\ldots n,$0 are then refined under the global prompt $b_i \subset \mathbb{R}^3,\quad i=1\ldots n,$1 by minimizing

$b_i \subset \mathbb{R}^3,\quad i=1\ldots n,$2

using a depth-conditioned ControlNet. The purpose of this stage is global texture coherence across the assembled scene.

The implementation settings reported for the experiments are specific. Scene graphs are obtained via GPT-4 and layouts via Chain-of-Thought prompting. Node-to-3D generation uses an MVDream prior, ISM guidance, 3000 iterations with progressive $b_i \subset \mathbb{R}^3,\quad i=1\ldots n,$3 px resolution, and the first 600 iterations have localization off. Harmonization is run for 70 iterations on meshes. These details indicate that GraLa3D is not a single-pass generator but a staged optimization framework in which global consistency is deferred until after node-wise and super-node-wise geometry formation.

6. Evaluation, results, and limitations

GraLa3D is evaluated against GraphDreamer and GALA3D using CLIP Score over 200 random views and a user study with 29 raters (Huang et al., 2024). The reported quantitative results are as follows:

Method	CLIP Score	User study
GraLa3D	0.308 avg	78.8% preferred
GALA3D	0.281	—
GraphDreamer	0.233	—

The qualitative findings state that GraLa3D produces correct spatial layouts and interactions up to 15 objects, with examples including mermaid-throne and rabbit-cake. The experimental evidence is therefore presented as supporting the claim that the method overcomes the identified limitations and generates complex 3D scenes closely aligned with text prompts.

The paper also records several limitations. First, the LLM may produce layouts that slightly contradict interactions, such as non-overlapping boxes for “holding,” and manual prompt correction is currently needed. Second, mesh quality depends on 3DGS-to-mesh conversion; future work could integrate stronger occupancy constraints or more efficient Gaussian-to-mesh approaches, with SuGaR mentioned as an example. Third, automatic LLM consistency checking or fine-tuning is proposed to further reduce manual intervention.

These limitations delimit the current scope of the method. GraLa3D tightly couples scene graphs, explicit layouts, and diffusion-based SDS guidance, augmented by interaction and localization losses. At the same time, the reported dependence on LLM-produced layouts and on downstream mesh extraction indicates that control fidelity at the semantic level and geometric fidelity at the mesh level remain partially decoupled, and that further work is anticipated at both interfaces.