SceneAssistant: Iterative 3D Scene Generation

Updated 5 July 2026

SceneAssistant is a visual-feedback-driven agent for 3D scene generation that iteratively creates and refines open-vocabulary layouts using rendered images.
It integrates a vision-language model, a 3D asset pipeline, and Blender-based APIs to enable precise object creation, manipulation, and camera control.
Empirical evaluations demonstrate superior spatial layout, object quality, and human preference compared to baselines in both indoor and open-vocabulary settings.

SceneAssistant is a visual-feedback-driven agent for open-vocabulary 3D scene generation that treats text-to-3D scene synthesis as an iterative tool-using process rather than a one-shot layout prediction problem. Given a natural-language description $d$ , it seeks to produce a 3D scene $s_d$ whose objects, layout, orientation, scale, and overall composition align with the prompt, while using rendered images as the primary source of truth during generation and editing. The framework combines a Vision-LLM (VLM), a 3D object generation pipeline, a Blender execution backend, and a compact set of scene-editing APIs, and it extends the same loop to scene editing through natural-language intervention (Luo et al., 12 Mar 2026).

1. Concept and scope

SceneAssistant addresses text-to-3D scene generation with an emphasis on open-vocabulary 3D scene synthesis. In the paper’s formulation, existing systems are limited because many are domain-restricted and many rely on a fixed vocabulary of spatial primitives such as on, in front of, against wall, or face to. SceneAssistant instead assumes that a description is not confined to a predefined domain such as indoor scenes and may include arbitrary objects and unconstrained configurations (Luo et al., 12 Mar 2026).

The system’s central premise is that visual feedback is essential for scene composition. Rather than predicting a complete scene once and committing to it, SceneAssistant mirrors a manual 3D workflow: create objects, inspect the rendered result, notice errors in placement or orientation, and revise the scene. This design is motivated by the instability of generative 3D assets and by the fact that correct composition depends on integrated scene appearance rather than on symbolic relations alone. A plausible implication is that SceneAssistant shifts scene generation from relation-graph solving toward iterative embodied editing over an explicit renderer.

2. Architecture and action interface

SceneAssistant is organized as a closed-loop, visual-feedback-driven agentic framework. At each step, the VLM receives the current rendered image, the user instruction, all previous actions, current scene data in JSON, and system messages such as warnings or edit instructions. It then emits a brief reasoning statement and a JSON list of Action API calls, which are executed in Blender before the scene is rendered again (Luo et al., 12 Mar 2026).

The action space is the system’s main control abstraction.

Category	APIs	Function
Object creation and removal	`Create`, `Duplicate`, `Delete`	Asset synthesis, cloning, deletion
Object manipulation	`Translate`, `Place`, `Rotate`, `Scale`	Relative or absolute pose and size control
Camera control	`ViewScene`, `FocusOn`, `RotateCamera`, `MoveCamera`	Global or local visual inspection
Environment and termination	`GenerateFloorTexture`, `Finish`	Floor appearance and stopping

Several prompt-level execution constraints shape this interface. Create and Duplicate must be executed in a separate batch, the agent should build incrementally, it can manipulate/create/duplicate at most 3 kinds of objects in one batch, and Finish() must appear alone. The coordinate convention is explicit: the floor is at $Z=0$ , $+Z$ is Up, $+X$ is Right, and $-Y$ is Forward (Luo et al., 12 Mar 2026).

This interface is functionally complete in the sense intended by the paper: if a VLM already has latent spatial reasoning ability, then arbitrary composition can be realized through sequences of Create, Place, Rotate, Scale, and camera-inspection actions, without relying on a predefined ontology of spatial relations.

3. Visual feedback loop and scene state

The core loop is ReAct-style. The scene starts empty, or from an existing scene in editing mode. After each action batch, Blender renders the updated scene, backend checks are applied, and the resulting image and messages are returned to the VLM. The process repeats until Finish() is called or a maximum step limit $T_M$ is reached; the implementation uses $T_M = 20$ (Luo et al., 12 Mar 2026).

The rendered image is treated as the authoritative scene state. The paper states that the system does not provide all previous images, because that would overload context, and it intentionally avoids carrying forward previous reasoning because earlier reasoning may contain errors. Instead, history is summarized through the action list plus the current scene JSON. This design privileges the current rendered scene over latent internal plans.

To improve grounding, SceneAssistant augments the render with two visual prompts: object semantic labels and a coordinate axis HUD. These overlays help the VLM localize objects and interpret manipulations in a stable reference frame. Backend rules provide additional structure: objects below the floor are automatically lifted to rest on $Z=0$ , and a BVH-based algorithm detects mesh collisions. By contrast, floating-object avoidance is not physically enforced and remains dependent on the VLM’s visual judgment (Luo et al., 12 Mar 2026).

4. Object generation and scene editing

SceneAssistant uses a generation-based asset pipeline rather than pure retrieval. The implementation pairs Gemini-3.0-Flash with Z-Image for image generation, Hunyuan3D for image-conditioned 3D asset reconstruction, and Blender for composition and rendering. The object workflow is: generate a high-quality object image from the text description, remove the background, reconstruct a textured mesh with Hunyuan3D, and import that mesh into Blender (Luo et al., 12 Mar 2026).

Entity extraction and relation interpretation are not delegated to a separate symbolic parser. The VLM itself decides what objects to create, how to name them, what descriptions to use in Create, and how many duplicates are required. Placement is handled by Place and Translate, orientation by Rotate, size by Scale, and inter-object relations are realized indirectly through absolute coordinates and iterative revision after visual inspection.

Editing uses the same loop as generation. Users can inject new requirements through system messages during the trajectory, and the VLM responds with the same action APIs. The paper describes this as especially useful near the end of generation, when a human may notice an omission that the VLM missed. An example mentioned in the text is adding plants to every desk. This suggests that SceneAssistant is not a separate edit model layered over a completed scene; editing is simply continued agentic scene construction under new constraints (Luo et al., 12 Mar 2026).

5. Empirical evaluation

The evaluation includes 8 indoor scenes and 22 open-vocabulary scenes, for 30 test cases in total, with each generated scene evaluated by 10 human participants. The indoor baselines are Holodeck and SceneWeaver; for open-vocabulary generation, the paper introduces two ablations, NoActionAPI and NoVisFeedback (Luo et al., 12 Mar 2026).

Human evaluation reports three measures: Spatial Layout Correctness, Object Quality, and Human Preference Rate. SceneAssistant outperforms both indoor baselines on all three metrics. On indoor scenes, Holodeck scores 4.475 for layout, 4.763 for object quality, and 6.25% preference; SceneWeaver scores 5.800, 6.150, and 36.25%; SceneAssistant scores 6.888, 6.950, and 61.25% (Luo et al., 12 Mar 2026).

The open-vocabulary setting isolates the importance of action APIs and visual feedback. NoActionAPI scores 7.005 for layout, 6.591 for object quality, and 35.91% preference; NoVisFeedback scores 6.255, 5.673, and 26.82%; SceneAssistant scores 7.600, 7.277, and 65.00% (Luo et al., 12 Mar 2026). The paper also reports qualitative ablations showing that removing object labels and the axis HUD reduces localization precision, while removing collision checks increases interpenetration.

These results indicate that the method’s gains do not come solely from using a strong VLM or a strong object generator. The combination of structured action APIs, rendered visual feedback, and lightweight symbolic backend checks is the experimentally supported unit.

6. Relation to earlier scene-assistant paradigms

SceneAssistant belongs to a longer line of systems that use language to control scenes, but it differs in both representation and control strategy. "SceneSeer" (Chang et al., 2017) extracted explicit textual constraints, inferred additional structure from a learned spatial knowledge base, and generated 3D scenes by selecting and arranging database models. SceneAssistant retains the iterative refinement spirit but replaces explicit spatial priors with a visual-feedback agent loop. This suggests a shift from symbolic scene completion toward image-grounded action planning.

It also differs from atlas-based conversational editors such as "Chat-Edit-3D" (Fang et al., 2024), which centered an LLM over a tool zoo and used Hash-Atlas to decouple 2D editing from 3D reconstruction. SceneAssistant does not map edits onto atlases; it composes meshes directly in Blender and uses rendering as feedback. By contrast with "3DSceneEditor" (Yan et al., 2024), which edits Gaussian scenes through instance segmentation and CLIP grounding, SceneAssistant does not assume a pre-reconstructed scene representation and instead generates assets on demand. Relative to object-centric simulation systems such as "ASSIST" (Zhong et al., 2023), which store each object as a scene node with a differentiable forward model and bounding box, SceneAssistant is less concerned with reusable neural scene assets and more concerned with open-vocabulary generative composition.

7. Limitations and significance

The paper is explicit about several limitations. Despite instructions to proceed incrementally, the agent often creates many objects first and then tries to adjust them all at once. It struggles on highly complex scenes, underuses camera tools beyond ViewScene and FocusOn, and depends on Hunyuan3D assets that can be malformed or low quality. The authors also note that the VLM may fail to determine even simple relations such as whether a chair faces a table, and satisfactory outputs may require repeated runs (Luo et al., 12 Mar 2026).

Within those limits, SceneAssistant’s significance lies in its control paradigm. It does not propose a new 3D representation or an explicit layout optimizer. Instead, it presents scene generation as a tool-using visual-agent problem: the agent creates, inspects, adjusts, and re-renders until the scene becomes coherent. In that sense, SceneAssistant is best understood as an agentic scene compositor whose main contribution is the integration of open-vocabulary 3D asset synthesis, direct manipulation APIs, and visual self-correction within a single iterative loop (Luo et al., 12 Mar 2026).