3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

Abstract: Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack. Authoring such procedural content, however, demands deep expertise in 3D software APIs, parametric design, and code-level geometric reasoning. In this paper, we propose 3DCodeBench, a systematic benchmark for evaluating vision-LLM (VLM) agents for procedural 3D generation in 3D modeling software. Specifically, 3DCodeBench evaluates how effectively 12 advanced VLMs can serve as procedural 3D modelers by translating text and image references into procedural code for 3D modeling software. Recognizing that automated metrics may not fully capture the perceptual quality of 3D shapes, we build 3DCodeArena, a ranking platform based on pairwise human preferences over generated 3D outputs. From extensive evaluations and results, we observe that: (1) Failures mostly arise from API mismatches, while successful renders still suffer from disconnected or floating 3D geometric components. (2) Test-time scaling, such as higher thinking budgets and multi-turn refinement, improves performance overall. Our findings highlight a critical need for high-quality procedural coding data to advance commercial VLMs. Furthermore, effective procedural 3D modeling requires a robust execution environment that provides high-fidelity feedback for iterative refinement. We release 3DCodeBench, including the curated large-scale dataset of multimodal (text/image) prompts, procedural code, 3D object triplets, evaluation protocol, and the public 3DCodeArena platform as a foundational toolkit for exploring VLM-based procedural 3D modelers.

• The paper presents a novel benchmark that systematically evaluates agentic procedural 3D modeling using both automated metrics and human preference assessments.
• It details an agentic data curation pipeline combining API migration, geometric validation, iterative multi-view feedback, and human-in-the-loop quality checks.
• Empirical results reveal that multi-turn error-feedback and model scaling significantly boost executability, with automated metrics showing strong correlation with human judgments.

3DCodeBench: A Comprehensive Benchmark for Agentic Procedural 3D Modeling

Introduction and Motivation

Procedural 3D modeling through code forms a cornerstone for asset creation in domains such as gaming, simulation, and digital design, due to its deterministic, editable, and scalable nature. However, authoring procedural assets requires deep knowledge of 3D APIs, parametric reasoning, and geometric logic. Existing efforts to automate this process via vision-LLMs (VLMs) and LLMs have outpaced standardized benchmarks for evaluating 3D code generation, leading to fragmented metrics and ad hoc test cases. "3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code" (2606.01057) directly addresses the deficit in systematic and robust evaluation for VLM-driven 3D procedural modeling, offering a comprehensive benchmark for agent-based code generation and assessment.

Figure 1: Overview of 3DCodeBench highlighting its benchmarking setup, semantic coverage, qualitative evaluation, and the 3DCodeArena human preference platform.

Data Curation Pipeline and Dataset Composition

The paper details a highly structured, agentic pipeline for curating the 3DCodeBench dataset, leveraging VLM agents to convert complex procedural factories sourced from Infinigen into standalone, API-accurate Blender 5.0 Python scripts. The pipeline incorporates API migration, geometric validation, and code simplification, augmented by iterative refinement using multi-view render comparison via a VLM-based visual critic. All (prompt, code, mesh) triplets are further validated by human annotators to guarantee semantic and geometric fidelity.

Figure 2: The data-curation pipeline, combining agentic transformation, geometric validation, iterative multi-view feedback, and human-in-the-loop quality inspection.

The resulting dataset captures substantial semantic and geometric diversity, spanning 212 categories—from organic entities to manufactured objects and architectural elements—with script lengths (mean 531 lines) reflecting code complexity not typically present in prior code-based 3D datasets.

Figure 3: Distributional and categorical statistics of 3DCodeBench, depicting code complexity and file size heterogeneity across curated scripts.

The procedural asset collection exceeds 26,000 (prompt, code, object) triplets, each annotated with multiple caption styles and paired multi-view imagery to support multi-modal evaluation protocols.

Evaluation Protocol: Automated Metrics and Human Preferences

3DCodeBench evaluates models not only for the executable validity of output scripts but also for mesh-grounded geometric similarity using a suite of automated metrics. These include, but are not limited to: executability, SigLIP-2 and DINOv3 perceptual embeddings for multi-view image similarity, Chamfer Distance for point cloud shape correspondence, and Uni3D for both 3D-3D and cross-modal (prompt-to-3D) metric alignment. Importantly, evaluation is stratified into conditional and penalized reporting paradigms to differentiate geometric quality from mere code execution reliability.

To anchor benchmark claims in human perception, 3DCodeArena conducts large-scale pairwise human preference comparisons, assigning Bradley–Terry Elo scores to each model.

Empirical Results and Analysis

Metric–Human Preference Alignment

A core empirical contribution is the demonstration that SigLIP-2 multi-view similarity and DINOv3 metrics exhibit strong linear and rank correlation with human Elo on 3DCodeArena (Pearson $r$ up to 0.964; Spearman $\rho$ up to 0.972), validating the use of automated metrics for large-scale evaluation.

Figure 4: Correlation between automated metrics (SigLIP-2, Executability, Uni3D, Chamfer Distance) and human Elo rankings across 12 evaluated VLMs.

Model Scaling, Agentic Refinement, and Multi-input Evaluation

• Thinking Budget: For lightweight models, increased reasoning budgets significantly uplift executability (e.g., +19 points for Gemini 3.1 Flash Lite), but this effect saturates for top-tier models, highlighting that model capacity, not merely inference tokens, governs ultimate performance.
• Multi-view Conditioning: Additional input views beyond a single canonical image offer negligible incremental gains on perceptual and geometric metrics for most models.
• Multi-turn Agentic Workflows: Multi-turn error-feedback loops dramatically elevate executability rates (aggregate from 0.692 to 0.972), with some models reaching the 1.0 ceiling. Shape quality as measured by SigLIP-2 and Uni3D also improves, though 3D-shape fidelity remains closely tied to model capacity.

  Figure 5: Ablation on thinking budget and input-view count across 12 VLMs, illustrating limited benefits from increased context for large models and saturation effects.

• Full Agent Harnesses: Wrapping VLMs in autonomous agentic harnesses increases executability further but does not yield higher shape quality on a per-instance basis, indicating the bottleneck has shifted from code correctness to fine-grained geometric reasoning.

Qualitative and Failure-Mode Analysis

Qualitative renders reveal that while current VLMs can roughly capture object silhouettes and part structure, they consistently struggle with connectedness, alignment, and physical plausibility—disconnects, floating parts, and geometric artifacts persist.

Figure 6: Qualitative outputs from Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.5, indicating proficiency in silhouette abstraction but challenges in achieving physically plausible, integrated geometry.

3DCodeArena: Human-in-the-Loop Ranking Platform

To address the limitations of automated evaluation, 3DCodeBench includes 3DCodeArena, a blind-pairwise voting interface supporting both text-to-3D and image-to-3D modalities. Anonymized, orbitable 3D renders are presented for direct geometric assessment, and aggregate Elo is calculated over thousands of human votes, enabling robust cross-model and cross-prompt comparisons.

Implications and Future Directions

This benchmark exposes gaps in both dataset coverage and model capacity: existing VLMs require curated, high-quality procedural code datasets for meaningful improvements and demonstrate that geometric reasoning and physical plausibility require further architectural and training advancements. The results underscore the importance of multi-turn refinement loops and high-fidelity feedback (e.g., error tracebacks, visual self-critique) for actionable progress. Furthermore, reliable automated metrics aligned with human judgment will be essential as the field transitions to larger-scale and cross-platform settings (e.g., Houdini, Unreal Engine).

Future developments may involve extending 3DCodeBench to scene-level composition, cross-platform API generalization, and more structured evaluation of multi-modal generative agents for complex 3D assets. The agentic methodology in data curation and evaluation protocol can be adapted for both foundational code LMs and embodied policy models in robotics and simulation.

Conclusion

3DCodeBench represents a significant step toward rigorous evaluation of procedural 3D model generation via agentic code synthesis. It provides (1) an extensible dataset capturing semantic and geometric diversity, (2) a robust, multi-faceted evaluation protocol combining executable, metric, and human-preference axes, and (3) empirical insights into the failure modes and improvement levers for current VLMs in 3D procedural modeling. The framework sets a foundation for the next generation of agentic 3D modelers capable of deterministic, high-fidelity, and physically plausible code-driven asset creation.