BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD

Abstract: Industrial Computer-Aided Design (CAD) code generation requires models to produce executable parametric programs from visual or textual inputs. Beyond recognizing the outer shape of a part, this task involves understanding its 3D structure, inferring engineering parameters, and choosing CAD operations that reflect how the part would be designed and manufactured. Despite the promise of Multimodal LLMs (MLLMs) for this task, they are rarely evaluated on whether these capabilities jointly hold in realistic industrial CAD settings. We present BenchCAD, a unified benchmark for industrial CAD reasoning. BenchCAD contains 17,900 execution-verified CadQuery programs across 106 industrial part families, including bevel gears, compression springs, twist drills, and other reusable engineering designs. It evaluates models through visual question answering, code question answering, image-to-code generation, and instruction-guided code editing, enabling fine-grained analysis across perception, parametric abstraction, and executable program synthesis. Across 10+ frontier models, BenchCAD shows that current systems often recover coarse outer geometry but fail to produce faithful parametric CAD programs. Common failures include missing fine 3D structure, misinterpreting industrial design parameters, and replacing essential operations such as sweeps, lofts, and twist-extrudes with simpler sketch-and-extrude patterns. Fine-tuning and reinforcement learning improve in-distribution performance, but generalization to unseen part families remains limited. These results position BenchCAD as a benchmark for measuring and improving the industrial readiness of multimodal CAD automation.

• The paper introduces BenchCAD, an execution-verified benchmark containing 17,900 CadQuery programs across 106 industrial part families, 46 CAD operations, and 47 standards codes to separately evaluate visual recognition, CAD-operation understanding, parametric abstraction, and spatial-code reasoning.
• The paper finds that current multimodal models perform substantially better on Code QA than Vision QA, while Vision2Code remains weak, with the best frontier model reaching only 0.397 and advanced-operation recall as low as 25.2%, exposing major gaps in faithful parametric program synthesis.
• The paper shows that fine-tuning can raise in-distribution Vision2Code performance to 0.7682 and operation recall to 84.0%, but out-of-distribution performance falls to 0.3776 and about 64% of apparently successful edits corrupt unrelated features, highlighting unresolved generalization and reliability problems.

Motivation and problem statement

BenchCAD addresses a specific gap in the evaluation of multimodal LLMs (MLLMs) for programmatic computer-aided design: existing benchmarks score only whether rendered output matches a target shape, typically via a single end-to-end metric such as IoU or Chamfer distance. The authors argue that this conflates three distinct capabilities — visual perception, parametric abstraction, and executable code synthesis — and can overestimate practical capability, since two programs with similar outer envelopes may differ substantially in editability, operation choice, and engineering detail. BenchCAD formalizes CAD reasoning as a four-level capability hierarchy ($L_1$ holistic visual recognition, $L_2$ CAD-operation understanding, $L_3$ industrial parametric abstraction, $L_4$ compositional spatial-code reasoning) and evaluates models against it.

Dataset construction

The benchmark comprises 17,900 execution-verified CadQuery programs across 106 named industrial part families (twist drills, bevel gears, compression springs, fasteners, flanges, brackets, etc.), each realized as an executable parametric program with STEP geometry, four canonical-view renders, parameter JSON, and an operation list. Four design principles govern construction:

• Expert-generated geometry: every family is hand-crafted from industrial standards, solving the standard-mandated geometric equations.
• Standard-table anchoring: 49% of families (52/106) sample parameters from real specification tables spanning 47 ISO/DIN/EN/ASME/IEC codes (e.g., DIN 338 drill diameters, ISO 23509 bevel-gear pitch–module relations).
• Family-level taxonomy: each (sub)family exposes a typed parameter schema, sampler, validator, and deterministic builder across easy/medium/hard difficulty tiers.
• Operation breadth: the corpus exercises 46 distinct CadQuery operations, including makeHelix, twistExtrude, polarArray, lofts, and parametric involute-gear construction — operations absent or rare in DeepCAD/Fusion360-derived corpora.

Verification is strict: programs are sandbox-executed and quarantined on parse errors, runtime exceptions, a 30 s timeout, or degenerate volume; surviving renders receive domain-expert visual sign-off. Two companion sets are released: BenchCAD-QA (2,400 paired image/code numeric questions, evaluated under both modalities) and BenchCAD-Edit (748 verified edit pairs stratified into five structural types T1–T5).

Task suite and metrics

Four task families decompose capability: Vision2Code (image-to-CadQuery generation, scored by voxel IoU, Chamfer/Hausdorff distance, feature F1, essential-op recall, execution rate), Code Edit (instruction-guided editing scored by headroom-normalized accuracy that clips improvement over the original-to-target IoU gap to $[0,1]$), and matched Vision QA / Code QA banks graded under $\pm5\%$ tolerance for ratios and exact match for integers. Blind baselines (blank images, blank code, unchanged programs) calibrate gains beyond dataset priors; notably, the blank-image baseline achieves 0.375 total Vision QA score, indicating substantial dataset priors in multiple-choice-free numeric answering.

Principal findings

Across 10+ frontier MLLMs and CAD-specialist baselines, several consistent patterns emerge:

• A large modality gap in QA. The best Code QA model reaches 0.838 total versus 0.587 for Vision QA on identical questions — a roughly 15–20 point "Holistic Spatial and Detailing Deficit" indicating that visual recognition, not reasoning over code, is often the bottleneck. Even with code access, spatial reasoning remains the weakest axis for most models (e.g., Gemini 3.1 Pro thinking scores 0.876 on parametric abstraction but only 0.537 on spatial reasoning).
• Vision2Code is far from solved. Frontier MLLMs cap mid-range (best frontier total 0.397, Gemini 3.1 Pro thinking); the specialist CADEvolve v3 reaches 0.6014 but with weak essential-op coverage (0.3715). Per-operation recall analysis shows macro recall of only 25.2% on advanced operations for GPT-5.3, with near-zero recall on chamfer, revolve, fillet, cboreHole, and shell — evidence of an "Operational Blindspot" where models substitute sketch-and-extrude patterns for sweeps, lofts, and twist-extrudes.
• Editing is harder than generation. All models trail on Code Edit relative to greenfield generation. Simple API-level edits (T1) are nearly solved (GPT-5.3 thinking: 0.865 aggregate), while compositional T5 rebuilds remain difficult. Replacing the textual instruction with a target render collapses all models to 0.04–0.34 mean-normalized score, since renders specify geometry but not numbers; augmenting text with the original render barely helps and even hurts GPT-5.3-thinking by up to 20 points on T5.
• Silent corruption of unrelated features. Approximately 64% of nominally successful edits silently corrupt non-targeted features ("Industrial Common Sense Gap"), a failure mode invisible to headline IoU. A case study shows three frontier models producing byte-different but geometrically identical wrong answers (IoU 0.961) on a through-bore edit, mistaking workplane-local .hole() for a global cut.
• Thinking modes are non-monotonic. Reasoning tiers lift capability up the L-hierarchy rather than uniformly reducing error: they close L1/L2 failures but expose an L3–L4 ceiling, and high input resolution can degrade thinking-model performance (total drops from 0.333 at 512 px to 0.275 at 768 px).

Training results and generalization limits

BenchCAD also functions as a training resource. Fine-tuning Qwen3-VL-2B yields the best in-distribution Vision2Code result (0.7682 total, 0.7520 IoU after SFT+RL), and GRPO-style RL with a reward combining essential-op recall and IoU raises execution rates above 98%. Operation-level recall improves dramatically over the pretrained baseline (macro recall over 16 operations rises from 17.5% to 84.0%), including zero-to-nonzero gains on revolve, fillet, loft, sweep, and helical sweeps. However, generalization is bounded: the OOD-SFT checkpoint scores 0.6064 IID but only 0.3776 OOD, and the strongest OOD performance comes from the IID-trained model — broad family coverage during supervised training, not RL, drives transfer. The authors state plainly that generalization to novel mechanical families requiring advanced operations and precise design constraints remains an open challenge.

Limitations acknowledged by the authors

The paper concedes three substantive limitations. First, parts are procedurally generated from expert-authored standard-parametric families rather than extracted from real engineering repositories, so proprietary design diversity (manufacturing tolerances, undocumented intent, assemblies) is not captured. Second, "standard-anchored" does not mean standard-compliant: families enforce inter-parameter relations from standards but do not validate full tolerance or material specifications. Third, the non-target preservation metric based on voxel IoU of the feature complement can miss subthreshold parametric shifts, which the authors partially mitigate with a complementary AST-diff metric. Additionally, all reported numbers are single-seed evaluations without error bars, a cost-driven concession the authors flag explicitly.

Conclusion

BenchCAD provides a unified, execution-verified, capability-decomposed evaluation framework for industrial CAD reasoning, combining scale (17,900 programs), standard anchoring (47 specification codes), operation breadth (46 CadQuery operations), and matched image/code contrastive tasks. Its central empirical contribution is diagnostic rather than leaderboard-oriented: it separates visual recognition deficits from operation-vocabulary gaps and parametric-abstraction failures, showing that current models recover coarse geometry but not faithful parametric programs, and that neither fine-tuning nor RL closes the out-of-distribution gap. The benchmark's open question is whether training signals grounded in standard-derived parameter relations can produce CAD priors that generalize beyond seen part families.