3Dify: Procedural 3D Generation Framework

• 3Dify is a procedural 3D-CG generation framework that leverages LLMs to translate natural language instructions into automated operations for DCC tools.
• The system employs a multi-agent architecture—with Visualizer, Planner, and Manager LLMs—and integrates RAG to interpret user intent and drive procedural 3D generation.
• It supports both MCP and CUA automation paths, enhancing interoperability and flexibility while addressing manual inefficiencies in complex 3D authoring workflows.

3Dify is a procedural 3D computer graphics generation framework that enables users to generate 3D-CG content solely through natural language instructions. Built upon Dify, an open-source platform for AI application development, it automates the operation of Digital Content Creation (DCC) tools such as Blender, Unreal Engine, and Unity, and incorporates several state-of-the-art LLM-related technologies such as the Model Context Protocol (MCP) and Retrieval-Augmented Generation (RAG) (Hayashi et al., 6 Oct 2025). The framework is designed to reduce the expertise and manual effort required for procedural modeling, especially the burden of constructing and maintaining large node graphs, while making the workflow more flexible across different tools and more adaptable to modern LLM ecosystems.

1. Definition and problem setting

3Dify was developed in response to a practical bottleneck in procedural 3D-CG authoring. Procedural generation is powerful because it can encode object structure, dependencies, and variations as rules and parameters rather than by hand-modeling every asset. In practice, however, users must translate diverse object variations into generation rules, implement those rules in often complex node graphs, and adapt that implementation across different DCC tools. The framework addresses the problem of how to generate procedural 3D content from natural language without requiring users to manually operate complex 3D software (Hayashi et al., 6 Oct 2025).

The system is positioned as a modular framework with four broad capabilities: it accepts natural-language instructions, uses an LLM-driven interactive image-refinement process to clarify intent, converts that intent into tool operations, and automatically drives DCC tools to produce the final 3D content (Hayashi et al., 6 Oct 2025). This organization reflects a procedural-authoring interpretation of “3Dify”: the central task is not direct end-to-end neural 3D synthesis, but orchestration of existing authoring systems through language-conditioned control.

A plausible implication is that 3Dify occupies a distinct systems layer within the current 3D generation landscape. Rather than replacing DCC software or procedural pipelines, it mediates between user intent and those pipelines, emphasizing interoperability, maintainability, and tool control.

2. System architecture

The architecture has four central parts: three LLM agents with distinct roles, RAG components referenced by the Planner LLM and Manager LLM, an image-generation feedback loop to refine the user’s visual intent, and automation interfaces for DCC tools through MCP servers and a fallback CUA mechanism when MCP is not available (Hayashi et al., 6 Oct 2025).

The three-agent division of labor is explicit. The Visualizer LLM generates candidate pre-visualization images from user instructions and regenerates candidates after user feedback. The Planner LLM examines the pre-visualization images, predicts what variability is needed in the 3D model, extracts procedural parameters and their ranges, and communicates a structured procedure to the Manager LLM. The Manager LLM receives the procedure from the Planner LLM, operates the DCC tool to create the 3D scene, and interacts with the user when needed (Hayashi et al., 6 Oct 2025).

Component	Role
Visualizer LLM	Generates and regenerates candidate pre-visualization images
Planner LLM	Extracts procedural parameters and communicates a structured procedure
Manager LLM	Operates the DCC tool and interacts with the user when needed

This separation is technically significant because it partitions intent discovery, procedural abstraction, and execution. The Visualizer handles ambiguity in user intent, the Planner maps visual targets to procedural structure, and the Manager handles tool-grounded action. This suggests an architecture aimed at reducing coupling between semantic interpretation and tool-specific execution.

3. Natural-language-to-3D workflow

The workflow is presented in three steps. First, the user provides a natural-language description of the desired 3D image or scene. Second, the LLM generates multiple 2D pre-visualization images, and the user selects the ones closest to the intended result; the system then regenerates new candidates based on those selections until the desired visual concept is sufficiently fixed. Third, the DCC tool is automatically operated to create the corresponding 3D content (Hayashi et al., 6 Oct 2025).

The image-generation feedback loop is described more concretely as follows. Suppose the user wants to generate $n$ images. The LLM first generates $n$ candidate images; these are low-quality, fast pre-visualization outputs. If they are too similar, the user can ask for more diversity. The user then selects $m \le n$ images that are closest to the target and can also explain why the others were rejected. The process repeats until $m = n$. Once the final $n$ candidates are sufficiently aligned with the user’s intent, the LLM generates high-quality 3D-CG images rather than preview images (Hayashi et al., 6 Oct 2025).

A key mechanism is that the LLM “automatically recognizes variable patterns” from the selected images and applies them to subsequent generations (Hayashi et al., 6 Oct 2025). Within the procedural framing of the system, this means that the framework infers latent dimensions of variation from preference signals rather than requiring explicit parameter editing by the user. This suggests a preference-driven route from informal visual specification to formalizable procedural variability.

4. Dify, Chatflow, and RAG as orchestration infrastructure

Dify functions as the orchestration backbone of 3Dify. The framework is built on Dify because it already provides a rapidly evolving LLM app framework, including model switching and workflow tools such as Chatflow. The paper emphasizes that using Dify gives 3Dify easy switching among LLM providers like OpenAI, Anthropic, and Google, open-source continuity, long-term maintainability, and rapid integration of new AI features (Hayashi et al., 6 Oct 2025).

Dify’s Chatflow is used to implement interactive multi-turn workflows and branching logic among agents. Chatflow has an important limitation: when returning to the start of a flow, information is reset unless it is explicitly stored in Conversation Variables. To support looping, branching, and agent switching, the system carefully stores state across turns. The paper lists the variables stage, dirty_bit, enable_increment, stage_num, and stages, with stages = ["Scene Analyzer", "RAG", "Conceptualization", "Builder", "Inspector"], and explains that TO_NEXT_STAGE increments stage_num automatically when dirty_bit == 0 (Hayashi et al., 6 Oct 2025).

For the Builder–Inspector loop, the framework introduces max_inspection_count and remaining_inspection_count to limit the number of inspection iterations (Hayashi et al., 6 Oct 2025). This stateful control structure is a notable part of the design because the framework is not merely a prompt wrapper around a single model; it is a multi-agent workflow system with explicit branching and bounded iterative refinement.

RAG is integrated in two main ways. First, it improves generation performance through Dify’s built-in RAG, specifically Parent–Child Indexing introduced in Dify v0.15.0. Second, it improves functionality and maintainability by allowing agents to reference manuals and documentation for DCC tools. The authors explicitly state that 3Dify does not limit RAG to tool-specific databases; they want the system to share procedural-generation concepts across tools, for example reusing knowledge about object placement constraints from Infinigen in Blender when working in Unreal Engine (Hayashi et al., 6 Oct 2025). This gives RAG a dual role: retrieval for immediate task performance and retrieval as a portability mechanism for procedural knowledge.

5. DCC automation through MCP and CUA

The main automation path is MCP. Each DCC tool has an MCP server, and the Manager LLM, via a Dify MCP client, sends tool-operation commands to that server, which then executes commands on the DCC application. The current implementation includes MCP servers for Blender and Unreal Engine, and the paper provides MCP server templates so other DCC tools can be added more easily (Hayashi et al., 6 Oct 2025).

A representative template includes a FastMCP server, a run_cmd_on_default_console(cmd: str) tool for executing commands in the DCC console, and a resource such as shortcut://keys to retrieve shortcut-key information (Hayashi et al., 6 Oct 2025). The framework describes MCP as the preferred path because it is straightforward, efficient, bidirectional, and good for structured tool use.

The paper also records limitations of MCP. As the number of MCP tools grows, the server becomes bloated; tool metadata and argument descriptions consume LLM context; and each function requires its own MCP tool implementation, which is labor-intensive for large DCC suites (Hayashi et al., 6 Oct 2025). These limitations motivate the fallback Computer-Using Agent (CUA) mechanism.

When a DCC tool cannot be accessed through MCP, 3Dify uses CUA automation. CUA allows the LLM to control the GUI directly from screenshots rather than through structured tool calls. The framework uses UI-TARS as its CUA-specialized model, described as a GUI agent fine-tuned from Qwen-VL and integrated through a custom Dify plugin using the SDK (Hayashi et al., 6 Oct 2025). CUA covers cases where MCP support is missing, the operation is too broad or too GUI-specific, or the tool functionality is difficult to expose as MCP endpoints. The trade-off is explicit: CUA is more flexible but has a higher chance of misoperation than MCP, so 3Dify sets a dynamic upper bound on the number of UI-TARS attempts depending on operation complexity (Hayashi et al., 6 Oct 2025).

If neither MCP nor CUA can perform a function, the framework requests manual user operation (Hayashi et al., 6 Oct 2025). This layered automation hierarchy—structured protocol, GUI automation, then manual fallback—indicates that the system is designed for heterogeneous production environments rather than for a single tightly controlled software stack.

6. Demonstration, limitations, and broader uses of “3Dify”

The paper includes a demonstration of the MCP-based automation path in Blender under Windows 11, Blender 4.2 LTS, Dify v1.3, a custom Dify MCP client v0.0.4, and the Blender MCP server ahujasid/blender-mcp. The demonstration prompt was: Create a desktop gaming PC model with side panel removed, keeping all internal components fully visible. The authors report that the desktop PC model was generated without directly operating Blender manually. Many MCP calls went through the code tool, which executes Blender scripts generated by the Manager LLM. A second demonstration showed that after multiple additional instructions, the system could successfully make the case fans glow. However, moving the entire PC upward did not work well: some parts protruded from the case (Hayashi et al., 6 Oct 2025).

The authors interpret this failure as a sign that spatial consistency becomes harder to maintain as the number of accumulated tasks grows and the LLM must track many objects and coordinates. They suggest that longer context windows may help and that CUA may be another promising direction. They also emphasize that this demonstration used MCP only, and the LLM did not incorporate visual feedback during 3D generation (Hayashi et al., 6 Oct 2025). Accordingly, the paper hints at several limitations: incomplete MCP coverage, context pressure from large tool sets, spatial consistency issues in long multi-step 3D editing sessions, lower reliability of CUA than MCP, and the absence of true visual feedback during generation (Hayashi et al., 6 Oct 2025).

In the broader literature, the term “3Dify” is also used more generally to denote conversion of non-3D inputs into usable 3D representations. Thing2Reality describes a system for “3Difying” images, sketches, web search results, screenshots, and camera-captured physical objects into conditioned multiview renderings or 3D Gaussians for XR communication (Hu et al., 2024). Edit3r is described as relevant to “3Dify-style workflows” because it makes the “2D edit $\rightarrow$ editable 3D scene” step fast enough for practical use while retaining cross-view consistency (Liu et al., 31 Dec 2025). 2D Instance Editing in 3D Space formulates a “2D-3D-2D” pipeline that lifts isolated 2D objects into a 3D representation, edits them in a physically plausible, rigidity-constrained 3D environment, and reprojects them back into the original image (Xie et al., 8 Jul 2025). Neuro-3D presents EEG-based 3D visual decoding as a direct step toward “3Dify” from brain signals by converting neural visual perception into 3D object reconstruction rather than 2D image synthesis (Guo et al., 2024).

Taken together, these uses suggest that “3Dify” has become a broader research shorthand for transforming language, images, edits, XR artifacts, or neural signals into interactive or editable 3D representations. Within that broader usage, the framework titled “3Dify: a Framework for Procedural 3D-CG Generation Assisted by LLMs Using MCP and RAG” is specifically a language-to-tool orchestration system for procedural 3D-CG authoring rather than a single 3D generative model (Hayashi et al., 6 Oct 2025).