GraphCanvas3D: 3D Scene Generation Framework

• GraphCanvas3D is a programmable, extensible framework for controllable 3D scene generation that uses hierarchical, graph-driven descriptions and in-context learning.
• It integrates LLM-based parsing, hierarchical graph optimization, and state-of-the-art 3D generative models to produce high-fidelity scenes with dynamic editing capabilities.
• The framework supports interactive scene manipulation and 4D temporal updates while ensuring global spatial consistency through modular, retraining-free processes.

GraphCanvas3D is a programmable, extensible framework for controllable 3D scene generation utilizing hierarchical, graph-driven scene descriptions and in-context learning. Spatial elements are modeled as nodes in a dynamically constructed scene graph, with edges encoding labeled spatial relations. The framework orchestrates LLM-based parsing, hierarchical graph optimization, and object synthesis via state-of-the-art 3D generative models, enabling on-the-fly scene manipulation—including 4D temporal dynamics—without retraining any neural modules (Liu et al., 2024).

1. System Architecture and Workflow

GraphCanvas3D implements a modular end-to-end pipeline that sequentially transforms a natural-language prompt into a high-fidelity rendered 3D scene. The pipeline comprises the following key components:

• LLM-Based Graph Constructor: Receives a scene text prompt $T_s$, parses it to identify object instances ($o_i$), generates per-object "node prompts," and infers directed edges (spatial relationships) using an LLM with a dedicated Prompt 1 template.
• Hierarchical Graph Optimizer: Optimization is performed at three levels:
  • Edge level: A multimodal LLM (MLLM) scores multi-view renders of paired objects, refining inter-object relations.
  • Subgraph level: Strongly connected nodes are grouped into subgraphs $G_i$ whose local layouts are refined via the optimizer.
  • Global level: Finalizes object placements across subgraphs to satisfy global consistency constraints.
• 3D Generative Module: Each object node is synthesized using an off-the-shelf point cloud generator (Point-E), upsampled, converted to a 3D Gaussian Splatting (3DGS) representation, and further refined with MVDream diffusion. The ensemble is composited in the final scene with ControlNet-guided rendering.
• In-Context Learning: All LLM or MLLM decisions are made via prompt engineering at inference time without parameter updates. Scene modifications and temporal dynamics are supported by re-parsing and local graph re-optimization using new prompts.

This design paradigm enables fully interactive, retraining-free scene creation, dynamic editing, and temporal (4D) scene evolution.

2. Hierarchical Scene Graph and Mathematical Foundation

The central representation is a hierarchical graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$:

• Nodes ($\mathcal{V}$): Each object $o_i$ possesses a feature vector

$\mathbf{f}_i = [x_i, y_i, z_i, s_i, r_i]$

comprising 3D position ($x_i, y_i, z_i$), isotropic scale ($s_i$), and yaw rotation ($r_i$).

• Edges ($o_i$0): Directed; each labeled by a spatial semantic such as "left of" or "on top of," associated with an edge-cost

$o_i$1

using position, rotation, and scale differentials.

• Subgraphs ($o_i$2): Local clusters of strongly interrelated nodes (enforcing mutual geometric constraints).
• Global graph ($o_i$3): Integration of all subgraphs subject to global constraints.

Scene graph optimization minimizes a global energy over both subgraph consistencies and inter-subgraph penalties: $o_i$4 with

$o_i$5

where $o_i$6 measures local coherence and $o_i$7 penalizes inter-subgraph misalignment.

Node positions are iteratively updated using gradients derived from multi-view relation losses: $o_i$8 where edge-level loss $o_i$9 aggregates weighted penalties from MLLM scores over multiple render views.

4D extensions index features temporally ($G_i$0), with temporal loss including coherence penalties

$G_i$1

to enforce plausible motion and consistency across frames.

3. Dynamic Scene Generation and Manipulation Algorithms

Scene construction and editing proceed according to the following schema:

$G_i$7

Dynamic Edits

• Addition: Add new node, infer new edges via LLM, perform local (edge and subgraph) optimization.
• Removal: Delete a node and its edges, re-optimize adjacent subgraphs only.
• Move/Reposition: Update feature vector $G_i$2, re-run only impacted edge and subgraph levels.
• 4D (temporal update): Provide a transformation description; LLM infers time-indexed node states, graph is optimized per frame.

4D sequences (e.g., an "apple rolls across the table") are managed by repeatedly applying LLM-guided transitions with temporal coherence regularization.

4. Empirical Evaluation and Benchmarks

GraphCanvas3D’s efficacy is assessed using both quantitative and qualitative measures. Core implementation is realized using ChatGPT-4o for both scene parsing and multi-view scoring. The generative stack comprises Point-E (for point cloud synthesis, 4K→100K point upsampling), 3D Gaussian Splatting for volumetric representation, and MVDream (diffusion-guided refinement, guidance scale 7.5). All experiments are conducted on a single NVIDIA A100 (24GB).

Quantitative Metrics:

• CLIP Score: Higher is better; gauges text–render alignment.
• MLLM Score: ChatGPT-4o ranking of multi-view scene consistency versus prompt.

Method	CLIP	MLLM
DreamGaussian	22.33	1.7
GaussianDreamer	26.17	3.0
MVDream	26.25	4.4
GS-Gen	26.28	4.1
GALA3D	28.67	7.0
GraphCanvas3D	29.67	8.3

User Study (n=67; 8 prompts):

Mean ratings (scale 1–10) for Scene Quality (SQ), Geometric Fidelity (GF), and Layout Realism (LR):

Method	SQ	GF	LR
DreamGaussian	5.22	4.18	4.30
GaussianDreamer	6.09	5.71	5.23
MVDream	7.32	7.98	7.07
GS-Gen	6.90	6.65	6.92
GALA3D	7.28	7.34	7.59
GraphCanvas3D	8.01	8.64	9.02

Qualitative Assessments:

• Multi-object scenes exhibit close visual/semantic correspondence to textual prompts.
• Edits (addition, deletion, movement), as well as temporal transitions, are handled with high fidelity and coherence.

5. Practical Usage: Programming Interface and Internal State

GraphCanvas3D is accessible via a unified Python interface, exposing explicit control over all scene graph and object-level parameters. Typical workflow:

$G_i$8

Internal Representation:

• Nodes:
  • o1 ("table"): $G_i$3
  • o2 ("apple"): $G_i$4
  • o3 ("cup"): $G_i$5
  • o4 ("book"): $G_i$6
• Edges:
  • e21: apple left_of table
  • e32: cup on table
  • e43: book right_of table

Scene rendering leverages four canonical views (front, side, top, oblique) for multi-view edge scoring and global consistency refinement.

6. Capabilities, Adaptability, and Unique Features

GraphCanvas3D provides a hierarchy-aware, controllable scene generation paradigm with the following distinguishing features:

• Retraining-Free Editability: All structural modifications (addition, deletion, repositioning, or temporal evolution of objects) are processed in real-time via in-context LLM/MLLM prompting; no neural weights are updated, ensuring rapid iteration.
• Hierarchical Control: Graph organization allows fine-grained, graph-theoretic manipulation down to pairwise inter-object spatial relations, as well as regionally coherent subgraph/group constraints.
• 4D Temporal Support: Direct handling of scene changes over time (including object transformations and movements) via time-indexed graph optimization.
• Model-Agnosticism: The framework is plug-and-play compatible with any LLM and 3D generator supporting the required interfaces.

A plausible implication is that GraphCanvas3D’s programmable graph abstraction, coupled with in-context adaptation, may serve as an extensible substrate for research in spatial intelligence, interactive 3D/4D editing, and embodied agency without necessitating retraining on new spatial tasks (Liu et al., 2024).