CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation

Abstract: Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existing evaluations are fragmented across datasets, modalities, and metrics. We introduce CADBench, a unified benchmark for multimodal CAD program generation. CADBench contains 18,000 evaluation samples spanning six benchmark families derived from DeepCAD, Fusion 360, ABC, MCB, and Objaverse; five input modalities including clean meshes, noisy meshes, single-view renders, photorealistic renders, and multi-view renders; and six metrics covering geometric fidelity, executability, and program compactness. STEP-based families are stratified by B-rep face count and all families are diversity-sampled to support controlled analysis across complexity and object variation. We benchmark eleven CAD-specialized and general-purpose vision-language systems, generating more than 1.4 million CAD programs. Under idealized inputs, specialized mesh-to-CAD models substantially outperform code-generating VLMs, which remain far from reliable CAD program reconstruction. CADBench further reveals three recurring failure modes: reconstruction quality degrades with geometric complexity, CAD-specialized models can be brittle under modality shift, and model rankings change across metrics. Together, these results position CADBench as a diagnostic testbed for measuring progress in editable 3D reconstruction and multimodal CAD understanding. The benchmark is publicly available at https://huggingface.co/datasets/DeCoDELab/CADBench.

• The paper introduces CADBench, a unified benchmark that addresses fragmented datasets and inconsistent metrics in AI-assisted CAD program generation.
• The study demonstrates significant performance differences between mesh-conditioned and image-conditioned models, highlighting trade-offs in accuracy and robustness.
• The benchmark’s design—including complexity stratification and multiple fidelity metrics—provides actionable insights for advancing CAD program synthesis.

CADBench: A Unified Multimodal Benchmark for AI-Assisted CAD Program Generation

Motivation and Problem Statement

Editable CAD program recovery from 2D or 3D observations remains a central task in computer-aided engineering, enabling downstream workflows that require precise, parametric, and modifiable design representations. Despite extensive recent progress in model architectures and generation paradigms for CAD code from multimodal inputs, the field lacks consistent, comprehensive, and diagnostic benchmarks. Existing evaluations are limited by narrow datasets, restricted operation vocabularies, fragmented modalities, inconsistent metrics, and little control over complexity or diversity. This fragmentation obscures model comparison, overestimates capabilities on easy examples, and prevents rigorous assessment of robustness to real-world variations.

CADBench addresses these limitations by introducing a large-scale, multimodal, and diagnostically structured benchmark suite for AI-assisted CAD program generation. It enables controlled analysis along axes critical for real-world deployment: geometric complexity, modality shift robustness, and metric-driven evaluation trade-offs.

Figure 1: CADBench evaluates eleven specialized and general-purpose CAD program generation models across 18,000 CAD samples, six benchmark families, five input modalities, and six primary metrics.

Benchmark Design: Structure, Modalities, and Complexity

CADBench organizes evaluation into six benchmark families that span canonical sketch-and-extrude problems, expanded operation sets, mechanical components, and organic or out-of-domain geometries. Families are sourced from DeepCAD, Fusion 360, ABC, MCB, and Objaverse, offering substantial coverage across real and synthetic engineering design regimes.

The STEP-based families (CAD-Base, CAD-Fusion, CAD-Extrude, CAD-All-Ops) employ face count stratification to produce explicit Low, Medium, and High complexity splits. This enables fine-grained measurement of degradation as geometric challenge increases, as face count aligns with established proxies for modeling difficulty. Mechanical and organic mesh-based families (CAD-Mechanical, CAD-Organic) are diversity-sampled without face-count stratification, recognizing the limitations of tessellation-centric complexity measurement for mesh data.

Figure 2: CADBench families are stratified across complexity levels and organized by modality, supporting targeted analysis by operation vocabulary and geometric detail.

Input modalities cover: (i) clean meshes, (ii) noisy meshes (Gaussian perturbed), (iii) single-view grayscale renders, (iv) multi-view stitched grayscale renders, and (v) photorealistic physically-based renders. This factorial design captures both the idealized and challenging conditions encountered in practical reverse engineering scenarios.

Figure 4: Each CadBench instance supports multiple input modalities, including clean/noisy meshes and diverse render variants.

Complexity and diversity sampling utilize DINOv3 latent embeddings and $k$-means clustering to construct representative and challenging splits, improving over random sampling by purposely increasing geometric and visual heterogeneity.

Evaluation Protocol: Metrics and Analytical Dimensions

CADBench advances benchmarking practice by adopting a comprehensive multi-metric evaluation suite:

• Geometric Fidelity: Volumetric IoU, Surface IoU (SIoU), and Chamfer Distance (CD) are measured after continuous Procrustes alignment to dissociate geometric error from orientation artifacts.
• Executability: Valid Shape Rate (VSR) quantifies the fraction of generated programs yielding valid solids, exposing pipeline brittleness.
• Program Compactness: Token and operation count serve as proxies for semantic parsimony and practical editability.

Geometric fidelity metrics are not redundant—IoU and SIoU provide only moderate correspondence ($\rho=0.45$), while CD is more strongly anti-correlated with either overlap metric (CD vs. IoU: $\rho=-0.62$; CD vs. SIoU: $\rho=-0.85$). This multidimensional approach is imperative, as model rankings are observed to shift based on which geometric axis is prioritized.

Figure 6: Correlation structure among geometric fidelity metrics, highlighting their non-redundancy and diagnostic complementarity.

Figure 8: Pairwise metric comparisons reveal that models may excel volumetrically while missing fine surface features, or vice versa.

Figure 10: Qualitative illustration of disagreement cases: high-IoU/low-SIoU masks missing detail, while high-SIoU/low-IoU can occur with thin-walled reconstructions.

Empirical Study and Key Findings

Eleven mesh- and image-conditioned models are benchmarked, spanning state-of-the-art CAD-specific architectures (CADFit, CADEvolve, CAD-Recode, Cadrille, CAD-Coder) and frontier vision-LLMs (Claude Opus 4.7, Gemini 3.1 Pro, GPT-5.4, Kimi K2.6, Qwen 3.5 9B/27B).

Aggregate Results: Under idealized conditions, mesh-conditioned models outperform all image-conditioned models on geometric metrics and execution robustness. For example, CADFit achieves $0.895$ IoU and $1.000$ VSR, whereas the top-performing image-based VLM, Claude Opus 4.7, attains $0.306$ IoU and $0.807$ VSR. General-purpose VLMs lag behind CAD-specialized approaches, and fail frequently on syntax and execution, underlining the gap between general code generation and domain-specific CAD program synthesis.

Complexity Sensitivity: Performance degrades monotonically as face count increases within sketch-and-extrude families. For most architectures, the drop between low and high complexity is severe, except for CADFit, which shows relative robustness, reflecting hybrid optimization pipelines. Diversity-aware sampling in splits further suppresses median scores compared to randomly sampled sets, indicating the inadequacy of random test splits for meaningful benchmarking.

Figure 3: Median IoU drops with face count, highlighting increased error on complex geometries for both mesh- and image-conditioned models.

Robustness to Modality Shift: Noise in mesh inputs and changes in renderings cause substantial performance drops in CAD-specialized models (e.g., CADEvolve $\Delta$IoU = $-0.523$ with noisy meshes). In contrast, VLMs show relatively stable, though lower, performance across image conditions. This suggests a trade-off between CAD-specific accuracy and generalization to real-world, noisy, or unfamiliar inputs.

Figure 11: Aggregate IoU across input variants exposes brittleness to mesh noise and rendering shifts, with generalist models more tolerant but consistently less accurate.

Metric Dependence in Model Ranking: Model ranking is non-invariant under different geometric metrics, reinforcing the necessity of multifaceted evaluation for CAD reconstruction. Aggregate leaderboards using a single metric may give a distorted sense of practical capability.

Implications, Limitations, and Future Prospects

Practical Implications: CADBench provides a rigorous standard for AI-driven 2D/3D-to-CAD program reconstruction, facilitating progress tracking and meaningful comparative analysis. It highlights the limitations of current generalist VLMs for structured design synthesis and the brittleness of specialized models to input distribution shift. This diagnostic insight is crucial for deploying AI-in-the-loop within engineering workflows, where stability and precision cannot be traded for raw generative flexibility.

Theoretical Implications: The findings emphasize the importance of explicitly modeling geometric complexity, data diversity, and robust input perturbations for progress in CAD program synthesis. The modular evaluation suite also motivates development of architectures tailored for both geometric accuracy and robust, cross-modality inference, as well as more refined program and semantic equivalence metrics.

Limitations: CADBench operationalizes success as the ability to synthesize an executable, valid solid with high geometric correspondence, not requiring explicit recovery of human-annotated operation sequences or constraints. This choice is necessitated by the available data, and future benchmarks may further extend evaluation to design intent and feature tree recovery. The VLM assessment does not employ prompt engineering or agentic refinement, which could modestly improve scores but would further complicate benchmarking standardization.

Future Directions: CADBench lays a foundation for expanding benchmarks to multi-turn pipelines, agentic correction mechanisms, semantic design correspondences, and real-world scanned input. As CAD data curation and AI model capabilities progress, the field can move toward evaluation protocols embracing design intent, constraint recovery, and manufacturability analyses, with robust cross-modal performance a key desideratum.

Conclusion

CADBench establishes a unified and diagnostically rich standard for benchmarking AI-assisted CAD program generation, providing explicit controls for geometric complexity, input modality, and evaluative dimensionality. The results elucidate current model deficiencies, the need for cross-modal and robust architectures, and the importance of multidimensional, fine-grained metrics in assessing practical and theoretical progress in CAD program synthesis. The benchmark’s public release may drive accelerated, rigorously measurable progress across the multimodal AI-assisted engineering design landscape.

(2605.10873)