SceneDesigner: Agentic Scene Synthesis

• SceneDesigner is a framework for controllable, multi-stage scene synthesis that integrates natural language prompts with executable asset generation.
• It employs hierarchical control and agent-driven modular design to enable precise, editable, and interactive 2D/3D world construction.
• By leveraging programmatic pipelines and error-recovery cycles, SceneDesigner ensures traceability and seamless integration with simulation and robotics applications.

SceneDesigner refers to a class of frameworks, methodologies, and systems enabling structured, controllable scene synthesis, spanning 2D controllable multi-object image generation, text-to-3D scene construction, interactive 3D world editing, and procedural or agentic scene assembly. Core research in this domain formalizes scene generation not merely as object arrangement or text-driven layout, but as an executable, traceable, and often editable pipeline encompassing natural language understanding, explicit state tracking, code-level asset generation, multi-modal reasoning, and hierarchical validation processes.

1. Architectural Principles and Agentic Pipelines

SceneDesigner systems consistently adopt multi-stage architectures decomposing scene synthesis into semantically meaningful roles and components. For instance, SceneCode ("Executable World Programs for Editable Indoor Scenes with Articulated Objects") organizes generation into a coupled two-module system: a room-level agentic backbone and a code-driven object generator (Wang et al., 19 May 2026):

• The Room-Level Agentic Backbone transforms natural-language prompts into a structured house layout and emits sequential per-object AssetRequests through a Planner–Designer–Critic loop. This LLM-driven agentic process orchestrates semantic stages (large furniture, wall-mounts, ceiling, manipulands), iteratively planning, proposing, and evaluating placements, while enforcing constraints (e.g., collision, style coherence).
• The object generator routes each AssetRequest to specialized code-generation profiles, synthesizing Blender Python programs for part-aware asset creation.

Other paradigms, such as MUSE ("Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction"), implement a closed-loop, multi-agent system consisting of Architect, Sculptor, and Inspector agents, operating on per-step, persistent, and reusable memory layers (Working/Scene/Skill Memory) to incrementally satisfy user-specified requirements while guaranteeing edit-locality and preservation (Xu et al., 12 Jun 2026). This aligns with FilmSceneDesigner’s finite-state chaining of specialized agents (manager, adjacency, check, shape, etc.) designed to mirror professional workflow, with explicit state handoff and structured parameter passing (Xie et al., 24 Nov 2025).

2. Programmatic, Executable, and Locally Editable Scene Representation

A central innovation in recent SceneDesigner frameworks is the elevation of scene structure from opaque mesh arrangements to executable, program-level representations. SceneCode embodies this by compiling each AssetRequest into a Blender Python program, which is validated, repaired, and registered under a unique object ID. This strategy yields direct local editability: parameter changes (e.g., altering a drawer’s depth) propagate solely to the affected code segment, allowing local regeneration and SDF re-export without global scene recomputation (Wang et al., 19 May 2026).

This programmatic paradigm is critical for traceability, downstream robotics, and simulation. By maintaining a persistent scene-state registry (house_state), the entire world-building process is rendered reproducible and modifiable, supporting inspection, object-level re-synthesis, and interactive augmentation.

3. Hierarchical Control, Object-Conditional Synthesis, and Validation

SceneDesigner systems address the heterogeneity of scene entities through explicit hierarchical control and strategy routing. SceneCode, for instance, employs a router to direct each AssetRequest to one of five code-generation strategies, accommodating the geometric and functional priors of wall art, static furniture, simple manipulands, structured manipulands, and articulated objects. Unique geometry and metadata (UVs, support face orientation, part independence for joints) are systematically embedded; articulated objects are exported with explicit joint schemas for physics simulation.

Both code and asset synthesis incorporate execution-guided repair-and-refine cycles, using automated error recovery (e.g., up to three runtime repairs and two critic-guided visual refinements) to guarantee code validity and semantic accuracy. This staged validation ensures only functionally correct, simulator-ready assets populate the final scene, in contrast to mesh-only or diffusion-based pipelines lacking deterministic correctness guarantees (Wang et al., 19 May 2026).

4. Evaluation Metrics, Empirical Performance, and Comparative Benchmarks

Quantitative and qualitative evaluation are core practices for SceneDesigner research, supporting systematic validation against established baselines. SceneCode delivers across scene-level and object-level metrics, including object count (CNT 79.4%), attribute match (ATR 74.0%), navigability (100%), collision fraction (11.3%), and human prompt-faithfulness relative to prior approaches (e.g., +2.8% versus SceneSmith, +24.6% versus LayoutVLM) (Wang et al., 19 May 2026). Mesh-level analysis reports zero non-manifold error, reduced face/vertex counts, and drastically cleaner UV islands as compared to state-of-the-art baselines.

Furthermore, assets generated with executable programs consistently provide better articulation metadata for manipulation benchmarks (linkage, joint type, pivot/axis, and limits), facilitating seamless integration into physics simulators like MuJoCo as demonstrated by executed open/slide robot actions.

5. Extensibility, Editability, and Downstream Applications

By representing the entire scene as executable world programs and linking all editable parameters to persistent registry entries, SceneDesigner frameworks support rapid iteration, partial regeneration, and sophisticated downstream use cases:

• Local and global editing: Direct parameter modification in object programs followed by localized script execution.
• Traceability: Full provenance of asset creation, placement, and style decisions accessible via object IDs.
• Robotics and embodied AI: Exported SDF/URDF with articulated kinematics supports manipulation policy evaluation and interaction studies out-of-the-box.
• Simulation and visualization: Physically plausible, collision-free, and style-coherent worlds readily rendered for user interaction or high-fidelity offline simulation.

These affordances distinguish SceneDesigner approaches from mesh aggregation or diffusion-image generation pipelines, establishing a foundation for interactive, high-fidelity, and purpose-driven world-authoring.

6. Comparative Landscape and Methodological Innovations

Compared to image-based controllable scene designers (e.g., multi-object 9-DoF image generation by SceneDesigner (Qin et al., 20 Nov 2025)), SceneCode extends the paradigm into physically-grounded, executable 3D world construction. Whereas prior text-to-3D frameworks focus primarily on object layout or surface mesh retrieval