MMGDreamer: Dual-Branch 3D Scene Diffusion

• MMGDreamer is a dual-branch diffusion framework that synthesizes geometry-controllable indoor scenes using mixed-modality graph representations.
• It leverages a visual enhancement module and relation predictor to infer and complete missing modality information, ensuring coherent scene layouts.
• Empirical evaluations show reduced FID, improved object-level metrics, and enhanced realism applicable to VR, interior design, and embodied AI.

MMGDreamer is a dual-branch diffusion framework for controllable 3D indoor scene generation that leverages mixed-modality graph representations, a visual enhancement module, and a relation predictor to synthesize geometry-controllable scenes from flexible multi-modal user inputs. The approach enables precise control over object geometry and scene layout, addressing limitations of prior text-based scene graph generators in adaptability and expressiveness, particularly in applications such as virtual reality and interior design (Yang et al., 9 Feb 2025).

1. Problem Definition and Motivation

Controllable 3D scene generation requires the synthesis of realistic scenes with explicit, user-dictated control over geometry and layout. Traditional graph-based methods typically accept only text inputs at the node level and rely on explicit user-specified object-object relationships, constraining input flexibility and limiting the capacity for fine-grained spatial control. MMGDreamer formulates indoor scene synthesis as a conditional diffusion process over graphs, wherein nodes may be specified by text, images, or both, and edge relations may be provided or inferred, thus enhancing adaptability to varying user input modalities and supporting direct geometry control (Yang et al., 9 Feb 2025).

2. Mixed-Modality Graph Representation

The central data structure is the Mixed-Modality Graph (MMG) $G^m = (V^m, R^m)$, with $N$ object nodes:

• Node attributes: Each $v_i^m$ may include a text token $o_i^m \in \mathcal{L}$ (e.g., "chair"), an image crop $i_i^m \in \mathbb{R}^{H \times W \times 3}$, or both. These are embedded as:
  • Category embedding $c_i^m \in \mathbb{R}^{d_c}$ (learnable)
  • CLIP-text embedding $$t_i^m = \textrm{CLIP}_\textrm{text}(o_i^m) \in \mathbb{R}^{d_t}$$
  • CLIP-image embedding $$u_i^m = \textrm{CLIP}_\textrm{image}(i_i^m) \in \mathbb{R}^{d_v}$$
  • Missing modalities are zero-padded.
• Edge attributes: $r^m_{i \rightarrow j}$ is embedded as an edge-type token $e^m_{i \rightarrow j}$, mapped to $N$0.

Node and edge features are collected as sets $N$1 and $N$2, respectively.

This design enables user inputs ranging from text-only descriptions to mixed text-visual specifications, with explicit or implicit (inferred) scene relationships.

3. Visual Enhancement and Relation Inference Modules

To address missing modality information, MMGDreamer incorporates the following modules:

• Visual Enhancement Module: For nodes specified by text only ($N$3), a VQ-VAE hallucinates plausible visual features $N$4 from the text embedding $N$5:
  • $N$6; quantized as $N$7; then decoded to $N$8
  • Training objective: ELBO with reconstruction and $N$9 terms,

  $v_i^m$0

• Relation Predictor: Infers missing or ambiguous object-object relations:

  • For each triplet $v_i^m$1, a GCN followed by an MLP predicts class scores for $v_i^m$2 relation types:

  $v_i^m$3 - Training loss: cross-entropy,

  $v_i^m$4

This facilitates coherent scene layouts even under incomplete user specifications.

4. Geometry Control via Mixed-Modality Inputs

By supporting real images $v_i^m$5 as part of node attributes (or via extraction from reference images using, e.g., GPT-4V), MMGDreamer "locks in" exact object geometries through CLIP-derived embeddings $v_i^m$6. The generative process utilizes these features to condition both the shape (object-level geometry) and layout (scene arrangement) diffusion branches. Box-level constraints, when provided (position $v_i^m$7, size $v_i^m$8, rotation $v_i^m$9), serve as hard initialization for the layout branch, ensuring the synthesis honors user-dictated geometry at multiple levels.

5. Dual-Branch Diffusion Architecture and Training Scheme

Scene generation proceeds via two coupled diffusion processes:

• Shape Branch: Models VQ-VAE latents $o_i^m \in \mathcal{L}$0 of object signed-distance fields.

  • Forward: $o_i^m \in \mathcal{L}$1
  • Reverse: Parameterized Gaussian $o_i^m \in \mathcal{L}$2
• Layout Branch: Models bounding box parameters $o_i^m \in \mathcal{L}$3 analogously.

At timestep $o_i^m \in \mathcal{L}$4, a graph encoder $o_i^m \in \mathcal{L}$5 (Triplet-GCN + echo mechanism) integrates noisy latents and node features (augmented via visual enhancement/relation prediction) into conditioned codes $o_i^m \in \mathcal{L}$6, which are fed to 3D-UNet and 1D-UNet denoisers for shape and layout, respectively. Denoising objectives are standard score-matching losses: $o_i^m \in \mathcal{L}$7

Training is staged: visual enhancement ($o_i^m \in \mathcal{L}$8) and relation predictor ($o_i^m \in \mathcal{L}$9) are trained separately, then fixed. The full end-to-end loss is

$i_i^m \in \mathbb{R}^{H \times W \times 3}$0

with coefficients determined by cross-validation (set to $i_i^m \in \mathbb{R}^{H \times W \times 3}$1 in practice).

6. Evaluation Metrics and Empirical Results

MMGDreamer is evaluated on the SG-FRONT benchmark for bedroom, living room, and dining room scenes, using both scene-level and object-level metrics:

Metric	Level	Description
FID, FID_CLIP	Scene	Fréchet distance on 256$i_i^m \in \mathbb{R}^{H \times W \times 3}$2 top-down renders, using Inception or CLIP features
KID	Scene	Kernel Inception Distance
MMD (Chamfer)	Object	Minimum Matching Distance with Chamfer metric
COV	Object	Coverage across generated and reference sets
1-NNA	Object	1-Nearest-Neighbor Accuracy for instance matching

MMGDreamer (MM+R), with both mixed-modality input and relation prediction, reduces FID by up to 9% and KID by up to 33% over the EchoScene baseline. Object-level geometry fidelity (MMD) is consistently improved and coverage (COV) increased across all ten evaluated object categories. Qualitatively, the method faithfully reproduces user-specified geometries, generates coherent layouts without explicit relations, and supports graph-level editing (node/edge changes) with over 0.90 consistency on key spatial relations ("left/right," "bigger/smaller," etc.) (Yang et al., 9 Feb 2025).

7. Applications and Significance

MMGDreamer addresses core limitations in controllable scene generation by decoupling modality constraints and tightly integrating geometry conditioning within a graph-diffusion framework. This supports a wider range of user input modalities and grants precise, direct control over both semantic and spatial properties. These properties enable applications in virtual reality, interior and architectural design, and data generation for embodied AI, where scene realism and flexible control at multiple abstraction levels are critical. The approach validates the efficacy of mixed-modality graphs and hierarchical diffusion for geometry-conditioned synthesis of complex 3D environments (Yang et al., 9 Feb 2025).