ShapeCraft: Procedural Text-to-3D Generation

• ShapeCraft is a multi-agent framework for text-to-3D generation that leverages a graph-based procedural shape (GPS) to produce structured, interactive assets.
• It employs hierarchical parsing and iterative multi-agent refinement to transform complex natural language into Blender-compatible geometry and texture programs.
• The framework outperforms comparable methods in alignment, topology accuracy, and runtime efficiency, enabling dynamic post-modeling interactions such as editing and animation.

Searching arXiv for ShapeCraft and related papers to ground the article. ShapeCraft is a multi-agent framework for text-to-3D generation that represents 3D assets as shape programs rather than as a single unstructured mesh or an implicit neural field. Its core representation, the Graph-based Procedural Shape (GPS), decomposes complex natural language into a structured graph of sub-tasks, after which LLM agents hierarchically parse user input to initialize GPS and then iteratively refine procedural modeling and painting to produce structured, textured, and interactive 3D assets (Zhang et al., 20 Oct 2025). Within the surrounding literature, ShapeCraft sits alongside CLIP-guided textured mesh optimization in ClipMatrix, recursive latent-shape evolution in ShapeCrafter, and grid-based LLM planning for clay deformation in LLM-Craft, but it is distinguished by its emphasis on structured geometry, editability, and post-modeling interaction (Jetchev, 2021, Fu et al., 2022, Bartsch et al., 2024).

1. Research setting and conceptual scope

ShapeCraft is introduced to address a specific limitation in text-to-3D generation: existing methods often yield unstructured meshes and exhibit poor interactivity, making them impractical for artistic workflows. Its stated goals are to produce well-structured geometry compatible with industry tools such as Blender, support post-modeling interactions such as shape editing and animation, and comprehend complex, long natural language descriptions with spatial relations and fine detail (Zhang et al., 20 Oct 2025).

The supplied literature distinguishes several shape-centric regimes. ClipMatrix starts from a rigged SMPL-X body model and optimizes mesh, texture, lighting, and camera parameters so that rendered images match CLIP text embeddings. ShapeCrafter instead maintains a latent 3D grid feature distribution that evolves as more phrases are added. LLM-Craft operates in a robotic setting, where a top-down image overlaid with a $4\times4$ grid is paired with discrete squeeze actions. ShapeCraft replaces these representations with procedural programs over component graphs (Jetchev, 2021, Fu et al., 2022, Bartsch et al., 2024, Zhang et al., 20 Oct 2025).

System	Representation	Update mechanism
ClipMatrix	SMPL-X mesh, per-vertex offsets, UV texture map	CLIP-based optimization through differentiable rendering
ShapeCrafter	VQ-DIF latent grid over T-SDF	Recursive phrase-conditioned refinement of $Z_t$
LLM-Craft	RGB image or binary $4\times4$ array	Iterative replanning over discrete squeezes
ShapeCraft	GPS shape program graph with Blender-Python snippets	Parser/Coder/Evaluator multi-agent refinement

This suggests that ShapeCraft should not be conflated with either single-step prompt-to-shape generation or purely optimization-based text guidance. In the summarized material, it denotes a program-structured approach whose central unit is the component-level procedural description rather than a monolithic mesh.

2. Graph-based Procedural Shape representation

ShapeCraft’s central data structure is a flat, rooted graph

$$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$

where $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$, with $v_0$ the virtual root node representing the whole object and $v_i$ the leaf nodes representing individual components. The edge set is a depth-1 star graph,

$\mathcal{E} = \{(v_i, v_0)\mid i>0\}.$

Each node $v_i$ carries four attributes,

$$\mathcal{A}(v_i) = \bigl(n_i^g,\; n_i^p,\; b_i,\; p_i\bigr),$$

where $Z_t$0 is a focused geometric description of the component, $Z_t$1 is a positional description, $Z_t$2 is a bounding volume $Z_t$3, and $Z_t$4 is an initially empty Blender-Python snippet that will generate the mesh (Zhang et al., 20 Oct 2025).

The initialization process is hierarchical parsing followed by flattening. The Parser agent reads the description $Z_t$5, constructs a conceptual decomposition, flattens that tree to leaf nodes only, instantiates each $Z_t$6, and populates $Z_t$7 and $Z_t$8. The Coder agent then uses $Z_t$9 to propose an initial bounding volume $4\times4$0, yielding a skeleton graph $4\times4$1 (Zhang et al., 20 Oct 2025).

To correct initial parsing or bounding-volume errors, ShapeCraft performs representation bootstrapping for $4\times4$2 rounds:

1. $4\times4$3
2. $4\times4$4

After $4\times4$5 iterations, empirically $4\times4$6, the method obtains a refined graph $4\times4$7 (Zhang et al., 20 Oct 2025). The significance of this design is that spatial relations and semantic shape details are externalized into explicit node attributes and bounding volumes before fine geometry is synthesized.

3. Multi-agent synthesis and iterative modeling

ShapeCraft comprises three LLM agents that share and update the GPS. The Parser agent decomposes the input into the GPS topology $4\times4$8 and generates $4\times4$9. The Coder agent fills in each node’s Blender-Python code snippet $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$0, both during GPS initialization and during later modeling. The Evaluator agent renders a partial or global shape via the procedural executor $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$1 and returns textual feedback and a scalar score,

$$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$2

The Coder agent leverages a thin wrapper library around the Blender API to constrain its action space to operations such as cube, sphere, cylinder, boolean operations, and bevel (Zhang et al., 20 Oct 2025).

The modeling loop uses multi-path sampling. For each leaf node $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$3, ShapeCraft creates $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$4 candidate paths, typically $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$5, and iteratively refines each one for up to $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$6 iterations. Early stopping occurs when the Evaluator score reaches a threshold $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$7. The best path is selected by

$$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$8

This procedure is formalized in the paper’s iterative shape modeling algorithm, in which the Coder proposes code, $$\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{A}),$$9 executes it, the Evaluator scores the rendered result, and the Coder revises the program using the feedback (Zhang et al., 20 Oct 2025).

A notable property of this pipeline is that geometry synthesis is component-local but globally conditioned. Each component is optimized as a node in $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$0, yet the Coder and Evaluator both receive the broader graph context. This suggests a mechanism for maintaining correct part arrangements while still allowing localized iteration over individual components.

4. Texturing, painting, and interactive editing

ShapeCraft extends procedural geometry with a learnable texture field

$\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$1

where $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$2 are BRDF parameters on the UV coordinate $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$3. Rendering is expressed as

$\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$4

under random view $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$5. Optimization uses Component-aware Score Distillation Sampling. The paper gives the SDS gradient as

$\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$6

and the final objective as

$\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$7

The first term uses the global text description $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$8; the second uses component-level prompts $\mathcal{V} = \{v_0\}\cup\{v_i\}_{i>0}$9 (Zhang et al., 20 Oct 2025).

Because each component is generated by a small Python snippet, artists can directly edit or extend the code. Post-modeling animations such as rotating a lamp shade or bending a robotic arm are implemented by prompting the same LLM agents on the existing GPS program. The public project page demonstrates a chat-style interface in which users can refine wording, add or remove parts, or request new animations on the fly (Zhang et al., 20 Oct 2025).

The framework also supports selective delegation. For very organic parts such as wings and tails, ShapeCraft can delegate to a specialized 3D API such as Hunyuan3D and then fit the result into its bounding box (Zhang et al., 20 Oct 2025). This indicates that the GPS representation functions not only as an overview target but also as an integration scaffold for heterogeneous geometry generators.

5. Evaluation and empirical profile

ShapeCraft is evaluated on a MARVEL-40M+ subset of 26 prompts. The reported metrics are Intersection-over-Ground-Truth (IoGT, higher is better), Hausdorff Distance (lower is better), CLIPScore (higher is better for text-3D alignment), VQA Pass Rate, Run Time, and API Calls (Zhang et al., 20 Oct 2025).

Method	Geometry	Alignment and cost
3D-PREMISE	IoGT 0.385; HD 0.527	CLIP 26.76; VQA 0.33; 2.81 min; 6 calls
CADCodeVerify	IoGT 0.334; HD 0.511	CLIP 25.94; VQA 0.34; 3.06 min; 9 calls
BlenderLLM	IoGT 0.455; HD 0.511	CLIP 26.99; VQA 0.43; 5.11 min; N.A.
LLaMA-Mesh	IoGT 0.346; HD 0.464	CLIP 25.72; VQA 0.28; 15.64 min; N.A.
MVDream	IoGT 0.427; HD 0.411	CLIP 26.84; VQA 0.42; 32.10 min; N.A.
ShapeCraft	IoGT 0.471; HD 0.415	CLIP 27.27; VQA 0.44; 11.68 min; 21 calls

On this benchmark, ShapeCraft achieves the best IoGT, the highest CLIPScore, and the highest VQA Pass Rate, while matching MVDream’s Hausdorff accuracy at substantially lower runtime than pure optimization. Qualitative comparisons in the paper report more plausible topology, correct part arrangements, and prompt-specific details such as rust spots and angled legs (Zhang et al., 20 Oct 2025).

The paper also reports two ablation trends. First, multi-path sampling with $v_0$0 and $v_0$1 provides the best balance of accuracy and runtime. Second, hierarchical parsing materially affects performance: removing hierarchy yields IoGT $v_0$2, HD $v_0$3, and VQA $v_0$4, whereas ShapeCraft reports IoGT $v_0$5, HD $v_0$6, and VQA $v_0$7 in that ablation table (Zhang et al., 20 Oct 2025). These results are consistent with the framework’s central claim that explicit decomposition improves downstream modeling fidelity.

6. Relation to earlier systems, limitations, and interpretive context

Several earlier systems illuminate what ShapeCraft changes. ClipMatrix creates textured 3D meshes controlled by text prompts by starting from a rigged SMPL-X human body model, attaching a $v_0$8 UV texture map, rendering randomized views with a PyTorch3D differentiable renderer, and optimizing a CLIP-based semantic loss plus mesh regularizers with Adam for 400–800 iterations. Its outputs remain animatable because the SMPL skeletal rig is preserved (Jetchev, 2021). ShapeCrafter instead builds on vector-quantized deep implicit functions, uses a fine-tuned BERT$v_0$9 text encoder and recursive conditioning over a latent grid $v_i$0, and introduces Text2Shape++, a dataset of 369K phrase-sequence→shape set examples. It reports that longer prompts benefit most and that as $v_i$1 grows, entropy $v_i$2 falls from $v_i$3 to $v_i$4 while inter-sample CD shrinks from $v_i$5 to $v_i$6 (Fu et al., 2022). LLM-Craft moves into robotic crafting of elasto-plastic objects, using a top-down RGB image overlaid with a $v_i$7 Cartesian grid and a discrete action space of $v_i$8 squeezes; iterative replanning improves letter-shaping performance from CD $v_i$9, EMD $\mathcal{E} = \{(v_i, v_0)\mid i>0\}.$0 to CD $\mathcal{E} = \{(v_i, v_0)\mid i>0\}.$1, EMD $\mathcal{E} = \{(v_i, v_0)\mid i>0\}.$2 (Bartsch et al., 2024).

The name similarity between ShapeCrafter and ShapeCraft can obscure a substantive difference. ShapeCrafter is a recursive text-conditioned 3D shape generation model over a VQ-DIF backbone, whereas ShapeCraft is a multi-agent framework based on procedural programs and component graphs (Fu et al., 2022, Zhang et al., 20 Oct 2025). This suggests that ShapeCraft is best understood as a structural and workflow-oriented re-framing of text-to-3D generation, not merely as another latent generative model.

ShapeCraft’s reported limitations are also explicit. Ambiguous or overly brief prompts can cause the Parser to miss parts or the Evaluator to provide misleading feedback. Organic or highly detailed geometry remains challenging because the Coder’s current library focuses on primitives and modifiers. Very large scenes would require hierarchical depth-$\mathcal{E} = \{(v_i, v_0)\mid i>0\}.$3 graphs and more advanced dependency resolution. The paper identifies several future directions: expanding the wrapper library to include native CAD/CAM primitives and custom subroutines, introducing multi-modal grounding with sketches or reference images, developing end-to-end tuning of the Coder and Evaluator agents to reduce iteration counts, and extending the framework to dynamic scene generation in which both shape and behavior are co-modeled (Zhang et al., 20 Oct 2025).