WorldCraft: Agentic 3D World-Building Framework

Updated 13 July 2026

WorldCraft is a modular, agent-based framework for controllable, photo-realistic 3D scene construction from natural language descriptions.
It incorporates specialized agents like ForgeIt for asset generation, ArrangeIt for spatial optimization, and a Trajectory Control Agent for dynamic animation.
The system demonstrates superior efficiency and fidelity, outperforming diffusion-based methods in terms of consistency, aesthetics, and user satisfaction.

WorldCraft is a name used in recent arXiv literature for several AI systems concerned with controllable world construction, but its most direct usage denotes a framework for photo-realistic 3D world creation and customization via LLM agents. In that formulation, users specify scenes, objects, layouts, and animations through natural language; a coordinator agent then orchestrates procedural asset generation, hierarchical layout optimization, and trajectory control to produce indoor and outdoor scenes that can be exported for rendering, including in Blender (Liu et al., 21 Feb 2025). Closely related usages include an agentic framework for creating an executable and visualizable AI Town from text and a trajectory-centric framework that extends interactive video world models from camera navigation to object manipulation, indicating that “WorldCraft” has become associated more broadly with agentic, spatially grounded, and interactive world-building systems rather than a single fixed architecture (Sun et al., 14 Jan 2026, Gu et al., 24 May 2026).

1. Terminological scope and research setting

The term appears across adjacent but distinct problem settings. One line focuses on photo-realistic 3D scene creation and customization with LLM agents; another focuses on executable AI Town-like environments; a third focuses on object manipulation in interactive video world models. A common misconception is therefore to treat WorldCraft as a single canonical model. The literature instead uses the name for multiple systems that share an emphasis on controllability, natural-language interaction, and explicit world state (Liu et al., 21 Feb 2025).

Name in literature	Problem setting	arXiv id
WorldCraft	Photo-realistic 3D world creation and customization via LLM agents	(Liu et al., 21 Feb 2025)
World Craft	Executable and visualizable AI Town via user textual descriptions	(Sun et al., 14 Jan 2026)
WorldCraft	Object manipulation in interactive video world models	(Gu et al., 24 May 2026)

Within this family, the 2025 WorldCraft system is distinguished by its modular agent-based pipeline for scene creation, customization, arrangement, and animation. Its objective is to democratize photorealistic virtual world construction for non-experts who would otherwise need to operate conventional 3D modeling software, while still allowing control over individual object attributes and scene layout (Liu et al., 21 Feb 2025).

2. Agent architecture and orchestration

The 2025 WorldCraft framework is organized as a modular agent-based system overseen by a GPT-4–based Coordinator Agent. The pipeline begins with natural-language user input, proceeds through task decomposition and coordination, calls asset-generation and layout agents, optionally invokes animation, and outputs assembled scenes for rendering. The Coordinator Agent manages dialogues, assigns sub-tasks, and supports stepwise customization through multi-turn interaction rather than a one-shot prompt interface (Liu et al., 21 Feb 2025).

Its principal specialized agents are ForgeIt, ArrangeIt, and a Trajectory Control Agent. ForgeIt handles precise procedural object generation. ArrangeIt determines spatial arrangements that balance ergonomic and aesthetic considerations. The Trajectory Control Agent converts conversational commands into camera and object movements. The framework is also compatible with off-the-shelf deep 3D generators, which the Coordinator Agent can call as alternative or complementary sources of assets, especially for stylized or artistic content (Liu et al., 21 Feb 2025).

This architecture makes clear that WorldCraft is not merely a text-to-3D generator. It is a coordinated system in which natural-language commands are translated into procedural code synthesis, constrained numerical optimization, API-based spatial reasoning, and animation commands. That decomposition is central to its claim of versatility across single-object customization and large-scale interior and exterior scene design (Liu et al., 21 Feb 2025).

3. ForgeIt and procedural asset generation

ForgeIt is the object-generation agent responsible for fine-grained, controllable asset creation. It interfaces with procedural generators, specifically Infinigen, through code synthesis, and it supports conversational editing of geometry, texture, and style. The framework presents this as an alternative to diffusion-based 3D generation, emphasizing avoidance of expensive mesh extractions, parameter-level control, and iterative refinement through dialogue (Liu et al., 21 Feb 2025).

A defining mechanism is ForgeIt’s “ever-growing manual via auto-verification.” Given a text description, ForgeIt synthesizes code to generate the object, renders the result, and submits the output to a critic LLM that checks correspondence between the desired attribute and the rendered object from 8 views. If the result is unsatisfactory, feedback is provided; if it is satisfactory, the successful case is added to the manual for future reference. This yields a self-improving knowledge base of generation strategies (Liu et al., 21 Feb 2025).

WorldCraft also supports asset enrichment by integrating external deep 3D generators such as Meshy, LRM, CLAY, and Instant3D. In the system description, these generators are invoked when procedural methods lack a needed asset class or when stylized content is preferred. The resulting pipeline combines procedural controllability with broader asset diversity, rather than committing exclusively to either procedural or learned 3D synthesis (Liu et al., 21 Feb 2025).

4. ArrangeIt, spatial optimization, and trajectory control

ArrangeIt is the layout-generation agent. It constructs object trees that encode hierarchical dependencies, such as books on shelves and shelves in rooms, and decomposes arrangement into multiple subproblems. Each subproblem is modeled as a constrained numerical optimization problem, where natural-language spatial relations are converted into objective terms and constraints and solved via simulated annealing using the Metropolis-Hastings criterion (Liu et al., 21 Feb 2025).

The optimization is formalized as

$\min_{\{\mathbf{p}_i, \mathbf{\theta}_i\}_{i=1}^n} L(\{\mathbf{p}_i, \mathbf{\theta}_i\}_{i=1}^n) = \sum_{j=1}^{m} \lambda_j L_j(\{\mathbf{p}_i, \mathbf{\theta}_i\}_{i=1}^n) \quad \text{subject to} \quad c_1, c_2, \ldots, c_k,$

where $\mathbf{p}_i = (x_i, y_i, z_i)$ denotes 3D position, $\mathbf{\theta}_i = (\theta_{ix}, \theta_{iy}, \theta_{iz})$ denotes orientation, $L_j$ are objective terms for ergonomic and aesthetic criteria, $\lambda_j$ are scalar weights, and $c_k$ are hard constraints such as no overlap or minimum distance thresholds (Liu et al., 21 Feb 2025).

To operationalize language-conditioned layout, ArrangeIt uses APIs for spatial relationships, including alignment(objects, axis), distance(objA, objB), symmetry(objects), and no_overlap(objA, objB). The system can enforce relations as hard or soft constraints, which allows it to translate instructions such as “the pool table should be in the center” into an explicit optimization problem rather than an informal prompt interpretation (Liu et al., 21 Feb 2025).

The Trajectory Control Agent extends the same natural-language interface to animation. Built upon ChatCam and using CineGPT for text-to-trajectory, it extracts scene-independent trajectory descriptions, computes 3D paths, anchors them to objects using bounding boxes, and auto-generates Blender-compatible animation commands. This supports camera motion and general object animation, including instructions such as “Make the drone fly over the city,” and places animation under the same conversational control regime as scene construction (Liu et al., 21 Feb 2025).

5. Evaluation and empirical findings

WorldCraft was evaluated quantitatively and qualitatively. The reported quantitative measures include user and GPT-4 ratings on a 1–10 scale for consistency to prompt, aesthetics, and functionality; CLIP Score for image–text correspondence; runtime; and ablations focused on manual construction style and arrangement hierarchy. Qualitative evaluation addressed visual, geometric, and functional fidelity (Liu et al., 21 Feb 2025).

The system is reported to outperform Holodeck and DreamScene in consistency, aesthetics, functionality, and efficiency, with the highest user and GPT-4 scores for all major criteria. The reported CLIP score for prompt-to-output similarity is 0.384. The paper also states that typical scene runtime is much reduced relative to diffusion-based approaches. ArrangeIt’s hierarchical optimization yields better layout and functional scores than LayoutGPT, and ForgeIt’s dynamic manual with auto-verification yields higher quality objects than static or no-manual variants in ablation studies (Liu et al., 21 Feb 2025).

The qualitative examples emphasize high-quality, fully-furnished houses and outdoor cityscapes, as well as the ability to add new objects, edit objects, re-lay scenes, or animate at any stage. A common misconception is that conversational scene generation necessarily sacrifices geometric or functional structure. The reported results argue instead for the usefulness of explicit optimization, procedural control, and modular coordination in maintaining functionality while preserving prompt consistency and aesthetic quality (Liu et al., 21 Feb 2025).

6. Relation to adjacent world-building systems

WorldCraft sits within a broader research shift from one-shot generation toward iterative, spatially explicit, and executable world-building. WorldSmith, for example, addressed fictional world visualization with multi-modal image generation through text input, sketching, region-based filling, tile-based hierarchical composition, and tree-structured editing history. Its studies reported that 11 of 13 participants preferred combining text with sketch or region tools, and it framed hierarchical and spatial prompting as more expressive than global text alone (Dang et al., 2023). This suggests a continuity between WorldSmith’s layered 2D co-creative editing and WorldCraft’s move toward modular 3D scene orchestration.

A parallel line appears in “World Craft: Agentic Framework to Create Visualizable Worlds via Text,” which targets executable AI Town-like environments rather than photo-realistic scene rendering. That system combines World Scaffold with the multi-agent World Guild, whose Enricher, Manager, Critic, and Artist transform rough textual descriptions into a structured world representation $\mathcal{G} = (M, A, L, P)$ . It reports CFR 0.94, RCS 0.88, OVD 7.13, and VSA-V 6.80, and is positioned against code agents such as Cursor and Antigravity as well as LLM baselines such as Qwen3 and Gemini-3-Pro (Sun et al., 14 Jan 2026). The shared emphasis is not identical output format but agentic decomposition of intent, layout, and correction.

Minecraft-centered generation research further broadens the context. DreamCraft generates functional 3D artifacts from free-form text using quantized NeRFs and differentiable functional constraints over block distributions and adjacency rules (Earle et al., 2024). World2Minecraft reconstructs real-world indoor scenes as editable Minecraft environments via 3D semantic occupancy prediction and reports a pipeline that is 7x faster and requires about 14x fewer build actions than building scenes from scratch (Zhang et al., 30 Apr 2026). Dream-Cubed trains 3D diffusion models directly on block IDs and supports inpainting and outpainting from user-authored blocks (Merino et al., 22 Apr 2026). These systems differ from WorldCraft’s photorealistic 3D scene pipeline, but all treat world generation as a problem of controllable, structured, and editable spatial synthesis.

Benchmarks and datasets reinforce the importance of this shift. MineAnyBuild evaluates open-world AI agents on executable spatial plan generation, spatial understanding, spatial reasoning, creativity, and spatial commonsense, and reports that even the best model achieves only approximately 41/100 overall while top mental-rotation accuracy is approximately 26.7%, barely above random at 25% (Wei et al., 26 May 2025). PLAICraft contributes over 10,000 hours of millisecond-aligned multiplayer Minecraft data across video, game audio, microphone audio, mouse, and keyboard actions, with evaluations for object recognition, spatial awareness, language grounding, and long-term memory (He et al., 19 May 2025). A plausible implication is that WorldCraft’s reliance on hierarchical decomposition, explicit optimization, and procedural verification reflects broader recognition that spatial intelligence in generative systems remains difficult when handled by language alone.

The later 2026 “WorldCraft” extends the name into interactive video world models by adding object-level trajectory actions to camera navigation through Normalized World Trajectory, Spatial-Pathway LoRA, and Trajectory-Anchored State Persistence (Gu et al., 24 May 2026). Taken together, these usages show the term’s evolution from scene creation to executable simulation scaffolds and finally to persistent object manipulation in autoregressive video worlds. Across these variants, the recurring theme is the replacement of undifferentiated prompting with structured control over assets, layouts, actions, and state.