SceneSmith: Simulation-Ready Indoor Scene Generation

• SceneSmith is a hierarchical agentic framework for generating indoor scenes using natural language prompts, emphasizing realism, physical plausibility, and compositional diversity.
• It employs multi-stage visual language model agents to design architectural layouts, furniture arrangements, and small-object populations through text-to-3D synthesis, dataset retrieval, and physical property estimations.
• Quantitative evaluations show significant improvements over baseline methods in object density, collision reduction, and semantic fidelity, supporting advanced benchmarking for robotic manipulation and embodied AI.

SceneSmith is a hierarchical agentic framework for generating simulation-ready indoor environments from natural language prompts, with a focus on realism, physical plausibility, and compositional diversity. Developed primarily for robotic manipulation research and embodied AI evaluation, SceneSmith integrates multi-stage visual LLM (VLM) agents to synthesize architectural layouts, furniture arrangements, and dense small-object populations, employing a tightly coupled pipeline of text-to-3D asset generation, dataset retrieval, and physical property estimation. Quantitative analysis demonstrates that SceneSmith produces environments vastly exceeding prior synthetic scene generators in object density, physical stability, and semantic richness, enabling advanced benchmarking and policy evaluation for home robots and interactive agents (Pfaff et al., 9 Feb 2026).

1. Hierarchical Agentic Scene Generation Pipeline

SceneSmith implements a three-stage generation protocol: architectural layout, furniture placement, and small-object population. At each level, specialized VLM-based agents operate according to a designer–critic–orchestrator paradigm, jointly optimizing for contextual relevance, physical realism, and prompt adherence.

Architectural Layout:

A global natural-language prompt $\mathcal{T}$ is parsed by a "layout agent" to establish the number of rooms ($M$), their geometric parameters ($\mathcal{G}_j$), and adjacency constraints. A critic refines these specifications, and candidate placements are evaluated via edge-attachment backtracking. The scoring combines adjacency satisfaction and geometric centering:

$$S_{\mathrm{local}} = S_{\mathrm{base}} + w_{\mathrm{adj}} \cdot (\#\text{adjacent satisfied}) - w_{\mathrm{dist}} \| \mathbf{c}_r - \mathbf{c}_{\mathrm{centroid}} \|_2$$

Global compactness and stability are also measured:

$$S_{\mathrm{compact}} = \frac{ \sum_r \mathrm{Area}(\mathcal G_r) }{ \mathrm{Area}(\mathrm{BBox}(\cup_r \mathcal G_r)) }$$

Furniture Placement:

Given per-room prompts $\mathcal{T}_j$ and geometry $\mathcal{G}_j$, a furniture agent iteratively proposes object additions and refinements. The critic evaluates layout, accessibility ("reachability" via Minkowski-sum based traversability for robots), and physical placement. A collision-free post-processing and gravity-settling operation minimize inter-object penetrations.

Small-Object (Manipuland) Population:

Surface-level prompts $\mathcal{T}_{j,k}$ are generated for support surfaces (tables, counters), spawning sub-agent sessions. Tools such as stack creation, container filling, and pile arrangement utilize both agent instruction and physics-based filters to ensure visual plausibility and physical stability. Final SNOPT-based minimization ensures non-penetration, and any object failing a five-second Drake gravity simulation is removed.

2. Agentic Framework and Toolchain

The agent hierarchy is composed of three coordinated roles:

• Designer: Proposes edits through structured tool calls (add, move, remove, rescale), acting on the current state with access to rendered previews and prompt context.
• Critic: Provides scalar (0–10) stage-specific assessments—realism, functionality, spatial balance, completeness, faithfulness—supplemented with textual rationales. Metrics differ by scene level.
• Orchestrator: Implements a refine-and-rollback loop by tracking state checkpoints, enforcing acceptance criteria, and determining when a stage attains sufficient quality to advance.

The framework exposes an API with at least 50 tool actions, including collision checking, snap-to-object alignment, gravity simulation, and reachability analysis. Working memory spans the most recent two agent turns and a sliding image window, with LLM-summarized histories per stage.

3. Asset Generation, Dataset Routing, and Physical Property Estimation

SceneSmith integrates three principal asset provisioning strategies:

Text-to-3D Synthesis:

Natural language object descriptions are processed by a GPT-Image1.5 pipeline to synthesize renders, which are segmented by SAM3 and meshed with SAM3D. Rigid body meshes undergo orientation canonicalization, normalization, and convex (V-HACD) decomposition.

Dataset Retrieval for Articulated Objects:

ArtVIP library retrieves articulated assets by filtering CLIP-embedding matches according to type and size, enabling kinetic realism for interactable agents.

Physical Property Estimation:

VLMs estimate mass, friction (by inferred material class), and inertia tensors from model renders. Uniform density assumption is applied for novelty objects, and canonical Z-up alignments are enforced. Thin coverings (e.g., rugs, posters) are synthesized as lightweight planar meshes with physically-based rendering (PBR) materials.

4. Quantitative Performance Metrics and Evaluation

SceneSmith environments contain $71.1 \pm 13.0$ objects per room (3–6× higher than baseline methods), with low collision rates (COL $1.2\% \pm 0.6\%$), high static equilibrium (STB $M$0), and significantly reduced mean penetration ($M$1 mm) and displacement after gravity-settling ($M$2 mm). Extensive human evaluation (205 participants, 3,051 valid trials) yields a $M$3 realism win rate and $M$4 prompt faithfulness, both with Cohen's $M$5, $M$6. Automated metrics such as object count (CNT), attribute recall (ATR), object-out-of-reach (OOR), object arrangement recall (OAR), accuracy (ACC), accessible navigation (NAV), and objects-out-of-bounds (OOB) also consistently demonstrate significant improvements over prior synthetic scene benchmarks (Pfaff et al., 9 Feb 2026).

5. Integration in Benchmarking and Policy Evaluation Pipelines

SceneSmith's generative pipeline is central to frameworks such as OSMa-Bench++, which leverages prompt-generated scenes for extensible benchmarking of semantic mapping and manipulation. The pipeline includes:

• LLM-driven prompt generation and diversity enforcement;
• SceneSmith scene synthesis (yielding Drake-format assets, semantics, and lighting rigs);
• Adaptation through a multi-stage converter handling semantic normalization, material/texture repair, shader fallback, floor unification, navigation mesh construction, and lighting control;
• Export to Habitat-compatible assets;
• Synchronized sequence generation (e.g., via HaDaGe) for robot and vision simulation;
• Integration with 3D semantic mapping methods (e.g., ConceptGraphs, BBQ) and prompt-aware VQA engines.

Controllable stress-testing is supported via parameterized object density, occlusion level, and dynamic lighting. Prompt-grounded VQA categories leverage availability of the original scene specification for direct measurement of semantic mapping fidelity, especially under manipulation-relevant corner cases (Kurkova et al., 26 May 2026).

Table: Key Stages in the OSMa-Bench++ SceneSmith-Driven Pipeline

Stage	Function	Output
LLM Prompt Gen	Generate & filter NL scene prompts	Diverse prompt set
SceneSmith Gen	Synthesize Drake-format scenes	Meshes, semantics, materials
Adaptation Layer	Convert, normalize, repair, prep for Habitat	Habitat scene folder
Sequence Gen	Camera/agent traverse, RGB-D sync	Frames, trajectory data
Sem. Mapping/VQA	3D semantic maps, prompt-grounded QA	Scene graphs, QA scores

6. Comparative Analysis and Adaptation in Adjacent Domains

Frameworks such as FilmSceneDesigner implement agent-based chaining and procedural tool pipelines for cinematic set design, evidencing close methodological convergence with SceneSmith. Critical steps include sequential agent communication for parameter assembly, in-context prompt strategies for role and style specification, and domain-specific asset retrieval (e.g., SetDepot-Pro dataset for film props and textures). Empirical evaluation highlights benefits of iterative correctness checks and targeted prompt engineering for semantic, stylistic, and layout accuracy—factors reflected in SceneSmith’s attainment of geometric and physical precision (Xie et al., 24 Nov 2025).

A plausible implication is that SceneSmith's modular, agentic paradigm facilitates rapid adaptation to alternate generative domains (film, architecture, simulation), contingent on domain-appropriate asset libraries, critics, and procedural modules.

7. Limitations and Prospective Extensions

Identified limitations include intermittent failures in the text-to-3D pipeline on thin or transparent objects, residual instability in single-agent physical estimation (e.g., support classification errors), and dependence on asset retrieval for articulated objects rather than generative synthesis. LLM-based evaluator agents introduce some nondeterminism, suggesting a potential benefit from hybrid symbolic-visual scoring systems for fully automated benchmarking. Prospective work includes extension to outdoor and non-residential environment synthesis and scale-up to corpora exceeding 3,000 rooms for reinforcement learning applications (Pfaff et al., 9 Feb 2026).

---

References:

• "SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes" (Pfaff et al., 9 Feb 2026)
• "OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes" (Kurkova et al., 26 May 2026)
• "FilmSceneDesigner: Chaining Set Design for Procedural Film Scene Generation" (Xie et al., 24 Nov 2025)