neuralCAD-Edit: Benchmark for Multimodal CAD Editing
- The paper introduces neuralCAD-Edit, the first expert benchmark assessing multimodal CAD editing using real communication signals and producing valid B-Rep outputs.
- It employs a controlled study with experienced CAD designers to capture integrated requests through speech, pointing, drawing, and live interaction while preserving temporal alignment.
- The evaluation reveals a significant gap between current AI models and human performance, highlighting challenges in 3D spatial reasoning and multimodal understanding.
neuralCAD-Edit is a benchmark for multimodal-instructed 3D CAD model editing built from expert human designers working in real CAD software. It evaluates whether AI systems can interpret how professional engineers actually request edits—through speech, pointing, drawing, and live interaction with the CAD model, not only typed text—and then produce a correct edited CAD model as a B-Rep stored as a .step file. The benchmark is presented as the first benchmark for editing 3D CAD models collected from expert CAD engineers, and as the first benchmark of this kind rather than a synthetic, text-only, or goal-image-conditioned evaluation setup (Perrett et al., 17 Apr 2026).
1. Definition, scope, and task setting
neuralCAD-Edit is explicitly an evaluation-oriented benchmark for CAD editing rather than a training corpus. The task is method-agnostic: the input is an existing B-Rep CAD model and a request that may be text-only or multimodal, and the output is an edited B-Rep CAD model, again as a valid .step file. The benchmark therefore evaluates the full chain of multimodal understanding, 3D spatial and temporal grounding, and executable CAD manipulation, rather than only captioning, retrieval, or shape similarity (Perrett et al., 17 Apr 2026).
The benchmark is designed around real engineering communication. The paper argues that, in practical CAD workflows, users do not merely describe a target object in text; they manipulate the model while speaking, point to regions with the mouse, sketch or draw over the interface, and align verbal references with changing viewpoints and on-screen gestures. neuralCAD-Edit preserves this temporal alignment and uses it as part of the benchmark input. This distinguishes the task from settings in which the model is given only a final rendered target or a short typed instruction (Perrett et al., 17 Apr 2026).
The benchmark also broadens the CAD domain relative to earlier task formulations. It is balanced across parametric and dumb models, and across single-body and assembly inputs. This matters because many prior CAD generation and editing systems focus on single components or symbolic construction sequences, whereas neuralCAD-Edit evaluates editing of real CAD artifacts in an industry-standard representation (Perrett et al., 17 Apr 2026).
2. Benchmark construction and dataset design
The benchmark was created through a contained, controlled study involving 10 consenting expert CAD designers with 8 to 13 years of mechanical engineering design experience. All data was captured in Autodesk Fusion, and all input CAD models were curated to remove personally identifying information and intellectual property. Source CAD models came from the Fusion360 Gallery Dataset. The final selected inputs have average complexity of 298 faces, 782 edges, and 504 vertices (Perrett et al., 17 Apr 2026).
The benchmark balances two axes of CAD model structure. The first is parameterisation: parametric models retain design history, whereas dumb models have no parametric history and expose only the current state. The second is construction: single-body versus assembly. The appendix filtering criteria are also explicit: parametric assemblies have 5–200 parametric operations and 2–30 components/bodies; parametric single-bodies have 5–200 operations and exactly 1 component/body; dumb assemblies have 0 parametric operations and 2–30 components/bodies; dumb single-bodies have 0 parametric operations and exactly 1 component/body (Perrett et al., 17 Apr 2026).
For each selected input model, the requestor created three independent edit requests: easy, medium, and hard, intended to take 2 minutes, 5 minutes, and 10 minutes respectively. For every request, the benchmark contains two edited outputs: the requestor’s own edited CAD model and a second edit by a different expert CAD user who saw the request video and input CAD model but not the requestor’s final result. This is a consequential design choice, because it makes ambiguity observable rather than suppressing it into a single canonical target (Perrett et al., 17 Apr 2026).
The full benchmark covers 4 modalities, 4 parameterisation/construction combinations, 3 difficulty levels, and 4 requests per combination, yielding 192 requests and 384 edits. Additional corpus statistics are also reported: 1.9 hours of request time, 28.4 hours of edit time, 30K logged events, 24K viewport renders associated with events, and 78 average events per edit. The paper does not report train/validation/test splits and instead presents neuralCAD-Edit as an evaluation-only benchmark (Perrett et al., 17 Apr 2026).
3. Modalities, logging, and annotation protocol
neuralCAD-Edit defines four request modalities. Text consists of a request typed into a text box with no model manipulation and no speech. Interactive consists of a request video with the user interacting with the CAD model, together with mouse pointing / gesturing and concurrent spoken instructions. Draw-temp extends the interactive mode with a drawing tool whose annotations disappear after a few seconds. Draw-static uses the same drawing tool, but the drawings remain visible until manually deleted or until the request ends (Perrett et al., 17 Apr 2026).
The benchmark logs substantially more than final input-output pairs. For requests it records screen recording video, speech, and selections. For edits it records viewport renders, selections, events, start and stop timestamps, the final model, and the model state throughout editing. The benchmark itself primarily uses the input CAD model, the request, and the final CAD model, but the richer process logs are part of the benchmark’s broader structure (Perrett et al., 17 Apr 2026).
The annotation and validation pipeline is also explicit. Every request and edit was manually reviewed by an expert CAD user, and 12% were rejected as “not up to standard.” Request videos were extracted at 720p and 30 FPS. Speech was transcribed automatically with Whisper, then converted into oracle transcriptions by manual checking and correction by the original requestor, followed by acceptance by another expert. 49% of requests required at least one transcription correction; the average character edit distance between automatic and final transcript was 3.2, and average transcript length was 82.7 characters. Both automatic and oracle transcripts were then aligned with WhisperX to obtain timestamps for every word (Perrett et al., 17 Apr 2026).
These design choices have methodological significance. The benchmark does not reduce multimodal CAD communication to a static prompt; it preserves the alignment between spoken references, viewpoint changes, cursor motion, and drawing. A plausible implication is that success on neuralCAD-Edit requires temporal grounding over CAD interaction, not only stronger image-language modeling.
4. Evaluation methodology and scoring logic
The benchmark uses both automatic and expert-human evaluation. The automatic metrics are Chamfer distance, Voxel IoU, DINOv2 similarity on rendered isometric views, and Validity, defined as the fraction of outputs that are valid CAD models. The paper explicitly notes a limitation of these metrics: they compare to the requestor’s edited result and may penalize alternate but valid edits when the request is ambiguous (Perrett et al., 17 Apr 2026).
Human evaluation is central. Evaluators do not compare candidate outputs to the requestor’s final model. Instead, they see the request and a standardized rendering package consisting of six orthographic views—top, bottom, front, back, left, right—plus one isometric view. They assign two scores: instruction-following and model-quality. Each is rated on a 1–7 rubric and normalized to 0–1 in reported tables. Every output is rated by 5 CAD experts, and the paper reports the median score (Perrett et al., 17 Apr 2026).
Acceptance is the primary benchmark statistic. A candidate is accepted only if it reaches the rubric threshold for acceptability on both axes: instruction-following 5+ and quality 5+. This is stricter than a single composite score and is intended to reflect practical deployability in CAD work rather than mere geometric resemblance (Perrett et al., 17 Apr 2026).
The benchmark also reports inter-rater reliability using the two-way random-effects Intraclass Correlation Coefficient with absolute agreement for the mean of five raters:
with
This indicates high reliability for the aggregated expert judgments (Perrett et al., 17 Apr 2026).
For foundation-model testing, the paper uses a CadQuery-based harness. Models can import the input .step CAD model, write Python CadQuery scripts, run those scripts, inspect stdout/stderr, visually inspect the CAD result, revise scripts iteratively, and stop when they decide the task is complete. The harness exposes operations including primitive creation, sketches, extrusions, revolutions, fillet and chamfer, booleans, and assembly modeling (Perrett et al., 17 Apr 2026).
5. Quantitative results and empirical findings
The headline finding is a large gap between current frontier models and expert humans. The best tested model, GPT 5.2, achieves 0.25 human acceptance, whereas the human baseline reaches 0.78. The absolute gap is
$0.78 - 0.25 = 0.53,$
which the paper highlights as a 53 percentage point shortfall for the strongest AI relative to expert CAD users (Perrett et al., 17 Apr 2026).
The same result reveals a frequent misconception in CAD benchmarking: high syntactic validity does not imply strong editing performance. GPT 5.2 attains Validity 0.99, yet its Human instruction score is 0.48, Human quality is 0.39, and Human acceptance is 0.25. By contrast, the human baseline records Validity 1.00, Human instruction 0.74, Human quality 0.66, and Human acceptance 0.78. The benchmark therefore separates file validity from expert-acceptable editing competence (Perrett et al., 17 Apr 2026).
The full reported figures are also informative. The human baseline attains Chamfer-dist 22, Voxel-IoU 0.76, DINO-sim 0.93, Validity 1.00, VLM acceptance 0.45, and Human acceptance 0.78. GPT 5.2 achieves Chamfer-dist 50, Voxel-IoU 0.57, DINO-sim 0.66, Validity 0.99, VLM acceptance 0.30, and Human acceptance 0.25. Gemini 3 Pro records Chamfer-dist 110, Voxel-IoU 0.30, DINO-sim 0.36, Validity 0.58, and Human acceptance 0.10. Claude Sonnet 4.5 records Chamfer-dist 54, Voxel-IoU 0.18, DINO-sim 0.25, Validity 0.42, and Human acceptance 0.05. The GT human requestor reaches Human acceptance 0.82 (Perrett et al., 17 Apr 2026).
The breakdown analyses expose several systematic patterns. Human-edited outputs from requests with drawing are scored highest, and humans benefit from the additional precision of drawn instructions. Frontier models do not similarly benefit from drawing-rich requests, which the paper interprets as a weakness in understanding simultaneous talking and drawing. Across difficulty levels, the human baseline performs similarly well on easy, medium, and hard requests, while GPT and Claude perform better on easier requests and degrade as difficulty increases (Perrett et al., 17 Apr 2026).
The benchmark also studies evaluator behavior. Human ratings of instruction-following and quality are strongly correlated, whereas VLM-based evaluators are less reliable. The paper reports that VLMs tend to overestimate AI edit quality and under-recognize high-quality human edits. Among automatic metrics, DINOv2 similarity is the best proxy to human judgment, though still imperfect (Perrett et al., 17 Apr 2026).
6. Position within CAD editing research, limitations, and significance
The paper explicitly contrasts neuralCAD-Edit with prior setups. It is distinguished from CAD-Editor, which defines text-based CAD editing over symbolic sketch-and-extrude sequences and trains on 120k synthesized triplets produced by an automated data-synthesis pipeline (Yuan et al., 6 Feb 2025). It also differs from CADMorph, which addresses geometry-driven parametric CAD editing by taking an original construction sequence and a target shape and performing an iterative plan–generate–verify loop without edit-triplet supervision (Ma et al., 12 Dec 2025). neuralCAD-Edit instead benchmarks editing requests collected from expert designers and preserves multimodal, temporally aligned communication over B-Rep CAD models. This suggests that the benchmark sits above several existing task formulations and can evaluate systems arising from different modeling paradigms.
The benchmark also has explicit limits. It is intentionally modest in size at 192 requests, because it is constructed as a high-quality expert benchmark rather than a large training dataset. Human requests remain ambiguous, so automatic comparison against a single requestor edit is necessarily incomplete. VLM-based grading is not yet reliable enough to replace expert evaluators. The paper therefore treats human experts as the gold-standard assessors of instruction-following and CAD quality (Perrett et al., 17 Apr 2026).
Its qualitative failure analysis identifies recurring AI weaknesses: compositional visual reasoning failures, relative 3D positioning errors, weak understanding of existing geometry, and poor exploitation of multimodal drawings. One example described in the paper is a drone edit in which Gemini and GPT identify rotors and arms and copy them the correct number of times, but place them incorrectly in 3D and then incorrectly conclude the task is finished. Claude is reported to sometimes give up editing and instead attempt to generate a model from scratch (Perrett et al., 17 Apr 2026).
The significance of neuralCAD-Edit lies in what it operationalizes. It evaluates not merely shape generation or symbolic sequence prediction, but realistic CAD editing from the forms of communication used by professional engineers: temporally interleaved speech, pointer gestures, drawing, and live CAD interaction. By showing that even the strongest tested model remains far below expert humans in human acceptance, the benchmark defines a technically demanding target for multimodal CAD assistants, tool-using foundation models, and future CAD editing systems (Perrett et al., 17 Apr 2026).
