MGRF: Multimodal Graph Rectified Flow

• MGRF is a generative modeling framework that uses multimodal graphs and deterministic rectified flow for fine-grained 3D scene synthesis.
• It employs a tri-branch architecture to decouple and control layout, shape, and texture, ensuring per-object fidelity and global style coherence.
• Empirical results show improved scene realism, object-level fidelity, and faster inference compared to traditional diffusion-based methods.

Multimodal Graph Rectified Flow (MGRF) is a generative modeling framework designed for fine-grained, style-consistent 3D scene synthesis, with explicit control over object geometry, texture, and spatial layout. The technique, introduced in "FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow" (Yang et al., 20 Mar 2026), leverages a tight-coupled rectified flow backbone conditioned on multimodal graphs to enforce both per-object fidelity and global style coherence in indoor scene generation.

1. Rectified Flow: Formalism and Learning Objective

The foundational element of MGRF is the rectified flow model, which eschews the conventional diffusion stochastic differential equation in favor of a deterministic linear interpolation between ground-truth data and a standard Gaussian prior. Given $N$ objects, data is denoted as $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$, with $\mathcal D_1 \sim \mathcal N(0, I)$ sampled from the prior. The forward trajectory $\mathcal D_t$ follows a time-indexed linear path:

$$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$

yielding a constant target velocity field:

$$v_\mathrm{target}(\mathcal D_t) = \frac{d}{dt}\mathcal D_t = \mathcal D_1 - \mathcal D_0.$$

A neural network $\Theta_\mathcal D(\mathcal D_t, \mathcal C_t, t)$ is trained to predict $v_\mathrm{target}$. Scene sampling integrates the learned vector field backwards in time, starting from Gaussian noise, using the update:

$$\dot{\mathcal D}_t = -\Theta_\mathcal D(\mathcal D_t, \mathcal C_t, t), \quad t:1 \rightarrow 0,$$

typically discretized using Euler integration.

Conditioning is achieved via "rectification" through graph coupling: at each time-step $t$, node-wise conditionings $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$0 are computed by a message-passing GCN (InfoExchangeUnit) operating over the multimodal scene graph, incorporating both static multimodal node embeddings and the current noisy states. The training loss is the mean-squared deviation from the target velocity:

$\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$1

2. Multimodal Scene Graph Representation

The target scene is modeled as a directed graph $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$2, where each node $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$3 aggregates:

• $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$4: a learned embedding encoding object category.
• $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$5: a CLIP embedding of the object label, or zero if unavailable.
• $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$6: a CLIP or DINO visual embedding of a reference view, or zero if missing.

Edges carry typed relational labels $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$7, encoding spatial or stylistic constraints (e.g., "left of," "same style as"). The graph features are encoded via an $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$8-layer triplet Graph Convolutional Network (triplet-GCN), which alternates edge-wise MLP-based message passing and node-wise aggregation:

• Edge update: $\mathcal D_0 = \{d_0^1, \dots, d_0^N\}$9
• Node update: $\mathcal D_1 \sim \mathcal N(0, I)$0

Final outputs $\mathcal D_1 \sim \mathcal N(0, I)$1 are used as node features or dynamic conditions.

3. Tri-Branch Scene Generation Architecture

MGRF operates a tri-branch generative design, decomposing the scene into layout, shape, and texture, with each branch instantiated via a specialized rectified flow model coupled to the multimodal scene graph.

3.1 Layout Branch

Layout synthesis models the arrangement of $\mathcal D_1 \sim \mathcal N(0, I)$2 objects as oriented 3D bounding boxes, $\mathcal D_1 \sim \mathcal N(0, I)$3. The rectified flow backbone consists of 16 transformer blocks. Conditioning is mediated by a LayoutExchangeUnit (a GCN variant), fusing per-node bounding box states and multimodal node features to yield time-dependent conditions $\mathcal D_1 \sim \mathcal N(0, I)$4. Sampling involves rectifying from Gaussian noise to the natural distribution over layouts.

3.2 Shape Branch

Each mesh $\mathcal D_1 \sim \mathcal N(0, I)$5 is voxelized, $\mathcal D_1 \sim \mathcal N(0, I)$6, and encoded by a 3D VQ-VAE ($\mathcal D_1 \sim \mathcal N(0, I)$7) into shape codes $\mathcal D_1 \sim \mathcal N(0, I)$8. The rectified flow model ($\mathcal D_1 \sim \mathcal N(0, I)$9) operates over these codes, exchanging shape information via the ShapeExchangeUnit along the scene graph to enable cross-object reasoning. Samples $\mathcal D_t$0 are decoded with $\mathcal D_t$1 to reconstruct object shapes.

3.3 Texture Branch

Given decoded geometry, feature volumes $\mathcal D_t$2 are extracted by rendering multi-view images and feeding these into a structured-latent VQ-VAE ($\mathcal D_t$3) to obtain texture codes $\mathcal D_t$4. A sparse-transformer rectified flow model ($\mathcal D_t$5), with TextureExchangeUnit for exchanging appearance information along the scene graph, is used to sample consistent object textures. Final meshes $\mathcal D_t$6 are reconstructed via $\mathcal D_t$7 and assembled into the 3D layout.

The tri-branch scheme ensures modular training and specialization while permitting joint reasoning across object properties and relationships.

4. Training Protocol and Evaluation

Training Settings

• Dataset: SG-FRONT, based on 3D-FRONT, encompassing 15 relation types and ~2,500 room scenes of varied types (bedroom, living room, dining room).
• Preprocessing steps:
  • Meshes voxelized: $\mathcal D_t$8.
  • Multi-view images rendered, with DINOv2 features extracted for $\mathcal D_t$9.
  • VQ-VAEs ($$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$0, $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$1) trained for both shape and texture: latent dimension 8, codebook size 1024.
  • Multimodal scene graph $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$2 constructed with CLIP, DINO (visual), and learned embeddings.
• Each tri-branch is trained separately with the rectified-flow loss $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$3.
• 20% random masking of node modalities during training for robustness.
• Optimization: AdamW; learning rate decayed from $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$4 to $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$5 and to $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$6 over 82,000 steps, batch size 196.
• Inference: Euler solver with $$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$7 steps, classifier-free guidance strength 5.0.

Evaluation Metrics

Aspect	Metric(s)	Scope
Scene-level realism	FID, FID$$\mathcal D_t = (1-t) \mathcal D_0 + t \mathcal D_1, \qquad t \in [0, 1]$$8, KID	Top-down renders
Object-level fidelity	MMD, Coverage (COV), 1-NNA	Individual objects
Controllability	CLIPScore (top-down), FPVScore	Text/graph input
Style consistency	FPVScore (cross-object)	Appearance
Human preference	Prompt adherence, layout, quality, style, overall
Efficiency	Inference time per scene (A100)

5. Empirical Results and Ablations

FlowScene, with MGRF, demonstrates improvements over established graph-conditioned and language-conditioned baselines. For retrieval-mode generation in bedrooms, FID decreases from 42.38 to 35.01 and KID from 1.72 to −0.34 compared to MMGDreamer. Object-level metrics show MMD halved for nightstands and lamps, with COV increasing by 40–45 points. MGRF achieves high precision in satisfying spatial and stylistic graph constraints, matching or surpassing baselines under edge alteration and node addition tests.

For style consistency, "same style as" relational pairs see an order-of-magnitude reduction in inter-shape Chamfer Distance (e.g., lamps in bedrooms: CD 30.07→0.19). Ablation results highlight that enabling InfoExchangeUnits in all three branches yields the best FID (32.76), compared to higher error with none (50.83). Switching the diffusion backbone to rectified flow in the shape branch provides the largest single-branch gain; applying rectified flow to both shape and layout achieves optimal performance.

Regarding robustness, multi-view training ensures stable test-time performance across multi-view and single-view scenarios (ΔFID < 0.1), whereas single-view training does not generalize to the multi-view test (ΔFID = +7.97). The architecture yields significantly faster scene synthesis: layout + shape in 6.8 s (vs. 45.3 s for MMGDreamer), with full texture at 37.4 s.

6. Architectural Characteristics and Limitations

The MGRF backbone enforces tight object coupling during sampling, supporting both per-object geometric control and holistic scene-level style coherence. Its deterministic, few-step generation pipeline exhibits lower variance and faster inference than diffusion models. Multimodal graph conditioning unifies textual, visual, and category inputs, enabling flexible integration from various sources (LLM-parsed text, image selection, hybrid input). The tri-branch architecture decouples layout, geometry, and appearance, simplifying both training and specialization.

However, current deployment is limited to synthetic indoor environments (3D-FRONT); generalization to real-world or outdoor domains remains untested. Scene quality is sensitive to the fidelity of the input scene graph, with missing or noisy relational edges potentially introducing artifacts in layout or appearance. Objects with ambiguous or single-view observations may be inaccurately reconstructed, particularly without rich multi-view priors.

A plausible implication is that robust extension of MGRF to broader scene types and noisier graph inputs may benefit from further enhancements in graph construction and multi-modal input handling.

7. Context and Implications

Multimodal Graph Rectified Flow, as embodied in FlowScene, represents a principled and efficient approach to graph-conditioned 3D scene generation, integrating rectified flow dynamics with cross-object reasoning. It achieves strong object-level and scene-level control, outperforms prior baselines in realism and style metrics, and supports high-speed inference. The decoupling of layout, geometry, and texture provides architectural flexibility for research and application in 3D scene synthesis domains. Further exploration is warranted for adaptation to real-world scene diversity and more robust scene graph representations (Yang et al., 20 Mar 2026).