---
title: 'neuralCAD-Edit: Benchmark for Multimodal CAD Editing'
url: https://www.emergentmind.com/topics/neuralcad-edit
type: topic
---

# neuralCAD-Edit: Benchmark for Multimodal CAD Editing

neuralCAD-Edit is a benchmark for **multimodal-instructed 3D CAD model editing** built from **expert human designers** working in real CAD software. It evaluates whether AI systems can interpret how professional engineers actually request edits—through **speech, pointing, drawing, and live interaction with the CAD model**, not only typed text—and then produce a correct edited CAD model as a **B-Rep** stored as a `.step` file. The benchmark is presented as the **first benchmark for editing 3D CAD models collected from expert CAD engineers**, and as the **first benchmark of this kind** rather than a synthetic, text-only, or goal-image-conditioned evaluation setup [2604.16170].

## 1. Definition, scope, and task setting

neuralCAD-Edit is explicitly an **evaluation-oriented** benchmark for CAD editing rather than a training corpus. The task is method-agnostic: the input is an existing **B-Rep** CAD model and a request that may be text-only or multimodal, and the output is an edited **B-Rep** CAD model, again as a valid `.step` file. The benchmark therefore evaluates the full chain of multimodal understanding, 3D spatial and temporal grounding, and executable CAD manipulation, rather than only captioning, retrieval, or shape similarity [2604.16170].

The benchmark is designed around real engineering communication. The paper argues that, in practical CAD workflows, users do not merely describe a target object in text; they manipulate the model while speaking, point to regions with the mouse, sketch or draw over the interface, and align verbal references with changing viewpoints and on-screen gestures. neuralCAD-Edit preserves this temporal alignment and uses it as part of the benchmark input. This distinguishes the task from settings in which the model is given only a final rendered target or a short typed instruction [2604.16170].

The benchmark also broadens the CAD domain relative to earlier task formulations. It is balanced across **parametric** and **dumb** models, and across **single-body** and **assembly** inputs. This matters because many prior CAD generation and editing systems focus on single components or symbolic construction sequences, whereas neuralCAD-Edit evaluates editing of real CAD artifacts in an industry-standard representation [2604.16170].

## 2. Benchmark construction and dataset design

The benchmark was created through a **contained, controlled study** involving **10 consenting expert CAD designers** with **8 to 13 years of mechanical engineering design experience**. All data was captured in **Autodesk Fusion**, and all input CAD models were curated to remove **personally identifying information** and **intellectual property**. Source CAD models came from the **Fusion360 Gallery Dataset**. The final selected inputs have average complexity of **298 faces**, **782 edges**, and **504 vertices** [2604.16170].

The benchmark balances two axes of CAD model structure. The first is **parameterisation**: **parametric** models retain design history, whereas **dumb** models have no parametric history and expose only the current state. The second is **construction**: **single-body** versus **assembly**. The appendix filtering criteria are also explicit: parametric assemblies have **5–200 parametric operations** and **2–30 components/bodies**; parametric single-bodies have **5–200 operations** and exactly **1 component/body**; dumb assemblies have **0 parametric operations** and **2–30 components/bodies**; dumb single-bodies have **0 parametric operations** and exactly **1 component/body** [2604.16170].

For each selected input model, the requestor created **three independent edit requests**: **easy**, **medium**, and **hard**, intended to take **2 minutes**, **5 minutes**, and **10 minutes** respectively. For every request, the benchmark contains **two edited outputs**: the requestor’s own edited CAD model and a second edit by a different expert CAD user who saw the request video and input CAD model but not the requestor’s final result. This is a consequential design choice, because it makes ambiguity observable rather than suppressing it into a single canonical target [2604.16170].

The full benchmark covers **4 modalities**, **4 parameterisation/construction combinations**, **3 difficulty levels**, and **4 requests per combination**, yielding **192 requests** and **384 edits**. Additional corpus statistics are also reported: **1.9 hours** of request time, **28.4 hours** of edit time, **30K** logged events, **24K** viewport renders associated with events, and **78** average events per edit. The paper does **not** report train/validation/test splits and instead presents neuralCAD-Edit as an **evaluation-only benchmark** [2604.16170].

## 3. Modalities, logging, and annotation protocol

neuralCAD-Edit defines four request modalities. **Text** consists of a request typed into a text box with **no model manipulation** and **no speech**. **Interactive** consists of a request video with the user interacting with the CAD model, together with **mouse pointing / gesturing** and **concurrent spoken instructions**. **Draw-temp** extends the interactive mode with a drawing tool whose annotations **disappear after a few seconds**. **Draw-static** uses the same drawing tool, but the drawings remain visible until manually deleted or until the request ends [2604.16170].

The benchmark logs substantially more than final input-output pairs. For requests it records **screen recording video**, **speech**, and **selections**. For edits it records **viewport renders**, **selections**, **events**, **start and stop timestamps**, the **final model**, and the **model state throughout editing**. The benchmark itself primarily uses the input CAD model, the request, and the final CAD model, but the richer process logs are part of the benchmark’s broader structure [2604.16170].

The annotation and validation pipeline is also explicit. Every request and edit was manually reviewed by an expert CAD user, and **12%** were rejected as “not up to standard.” Request videos were extracted at **720p** and **30 FPS**. Speech was transcribed automatically with **Whisper**, then converted into **oracle transcriptions** by manual checking and correction by the original requestor, followed by acceptance by another expert. **49%** of requests required at least one transcription correction; the average character edit distance between automatic and final transcript was **3.2**, and average transcript length was **82.7 characters**. Both automatic and oracle transcripts were then aligned with **WhisperX** to obtain timestamps for every word [2604.16170].

These design choices have methodological significance. The benchmark does not reduce multimodal CAD communication to a static prompt; it preserves the alignment between spoken references, viewpoint changes, cursor motion, and drawing. A plausible implication is that success on neuralCAD-Edit requires temporal grounding over CAD interaction, not only stronger image-language modeling.

## 4. Evaluation methodology and scoring logic

The benchmark uses both automatic and expert-human evaluation. The automatic metrics are **Chamfer distance**, **Voxel IoU**, **DINOv2 similarity** on rendered isometric views, and **Validity**, defined as the fraction of outputs that are valid CAD models. The paper explicitly notes a limitation of these metrics: they compare to the requestor’s edited result and may penalize alternate but valid edits when the request is ambiguous [2604.16170].

Human evaluation is central. Evaluators do **not** compare candidate outputs to the requestor’s final model. Instead, they see the request and a standardized rendering package consisting of six orthographic views—**top**, **bottom**, **front**, **back**, **left**, **right**—plus one isometric view. They assign two scores: **instruction-following** and **model-quality**. Each is rated on a **1–7** rubric and normalized to **0–1** in reported tables. Every output is rated by **5 CAD experts**, and the paper reports the **median** score [2604.16170].

Acceptance is the primary benchmark statistic. A candidate is accepted only if it reaches the rubric threshold for acceptability on **both** axes: **instruction-following 5+** and **quality 5+**. This is stricter than a single composite score and is intended to reflect practical deployability in CAD work rather than mere geometric resemblance [2604.16170].

The benchmark also reports inter-rater reliability using the **two-way random-effects Intraclass Correlation Coefficient with absolute agreement** for the **mean of five raters**:  
\[
\mathrm{ICC}(2,k) = 0.88
\]
with  
\[
95\% \ \mathrm{CI}: [0.87, 0.89].
\]
This indicates high reliability for the aggregated expert judgments [2604.16170].

For foundation-model testing, the paper uses a CadQuery-based harness. Models can import the input `.step` CAD model, write **Python CadQuery** scripts, run those scripts, inspect **stdout/stderr**, visually inspect the CAD result, revise scripts iteratively, and stop when they decide the task is complete. The harness exposes operations including **primitive creation**, **sketches**, **extrusions**, **revolutions**, **fillet and chamfer**, **booleans**, and **assembly modeling** [2604.16170].

## 5. Quantitative results and empirical findings

The headline finding is a large gap between current frontier models and expert humans. The best tested model, **GPT 5.2**, achieves **0.25** human acceptance, whereas the **human baseline** reaches **0.78**. The absolute gap is  
\[
0.78 - 0.25 = 0.53,
\]
which the paper highlights as a **53 percentage point** shortfall for the strongest AI relative to expert CAD users [2604.16170].

The same result reveals a frequent misconception in CAD benchmarking: high syntactic validity does not imply strong editing performance. **GPT 5.2** attains **Validity 0.99**, yet its **Human instruction** score is **0.48**, **Human quality** is **0.39**, and **Human acceptance** is **0.25**. By contrast, the **human baseline** records **Validity 1.00**, **Human instruction 0.74**, **Human quality 0.66**, and **Human acceptance 0.78**. The benchmark therefore separates file validity from expert-acceptable editing competence [2604.16170].

The full reported figures are also informative. The **human baseline** attains **Chamfer-dist 22**, **Voxel-IoU 0.76**, **DINO-sim 0.93**, **Validity 1.00**, **VLM acceptance 0.45**, and **Human acceptance 0.78**. **GPT 5.2** achieves **Chamfer-dist 50**, **Voxel-IoU 0.57**, **DINO-sim 0.66**, **Validity 0.99**, **VLM acceptance 0.30**, and **Human acceptance 0.25**. **Gemini 3 Pro** records **Chamfer-dist 110**, **Voxel-IoU 0.30**, **DINO-sim 0.36**, **Validity 0.58**, and **Human acceptance 0.10**. **Claude Sonnet 4.5** records **Chamfer-dist 54**, **Voxel-IoU 0.18**, **DINO-sim 0.25**, **Validity 0.42**, and **Human acceptance 0.05**. The **GT human requestor** reaches **Human acceptance 0.82** [2604.16170].

The breakdown analyses expose several systematic patterns. Human-edited outputs from requests with **drawing** are scored highest, and humans benefit from the additional precision of drawn instructions. Frontier models do **not** similarly benefit from drawing-rich requests, which the paper interprets as a weakness in understanding **simultaneous talking and drawing**. Across difficulty levels, the **human baseline** performs similarly well on easy, medium, and hard requests, while **GPT** and **Claude** perform better on easier requests and degrade as difficulty increases [2604.16170].

The benchmark also studies evaluator behavior. Human ratings of instruction-following and quality are strongly correlated, whereas VLM-based evaluators are less reliable. The paper reports that VLMs tend to **overestimate AI edit quality** and **under-recognize high-quality human edits**. Among automatic metrics, **DINOv2 similarity** is the best proxy to human judgment, though still imperfect [2604.16170].

## 6. Position within CAD editing research, limitations, and significance

The paper explicitly contrasts neuralCAD-Edit with prior setups. It is distinguished from **CAD-Editor**, which defines **text-based CAD editing** over symbolic **sketch-and-extrude** sequences and trains on **120k synthesized triplets** produced by an automated data-synthesis pipeline [2502.03997]. It also differs from **CADMorph**, which addresses **geometry-driven parametric CAD editing** by taking an original construction sequence and a target shape and performing an iterative **plan–generate–verify** loop without edit-triplet supervision [2512.11480]. neuralCAD-Edit instead benchmarks editing requests collected from expert designers and preserves multimodal, temporally aligned communication over **B-Rep** CAD models. This suggests that the benchmark sits above several existing task formulations and can evaluate systems arising from different modeling paradigms.

The benchmark also has explicit limits. It is intentionally modest in size at **192 requests**, because it is constructed as a high-quality expert benchmark rather than a large training dataset. Human requests remain ambiguous, so automatic comparison against a single requestor edit is necessarily incomplete. VLM-based grading is not yet reliable enough to replace expert evaluators. The paper therefore treats human experts as the gold-standard assessors of instruction-following and CAD quality [2604.16170].

Its qualitative failure analysis identifies recurring AI weaknesses: **compositional visual reasoning failures**, **relative 3D positioning errors**, weak understanding of existing geometry, and poor exploitation of multimodal drawings. One example described in the paper is a drone edit in which Gemini and GPT identify rotors and arms and copy them the correct number of times, but place them incorrectly in 3D and then incorrectly conclude the task is finished. Claude is reported to sometimes give up editing and instead attempt to generate a model from scratch [2604.16170].

The significance of neuralCAD-Edit lies in what it operationalizes. It evaluates not merely shape generation or symbolic sequence prediction, but realistic CAD editing from the forms of communication used by professional engineers: **temporally interleaved speech, pointer gestures, drawing, and live CAD interaction**. By showing that even the strongest tested model remains far below expert humans in **human acceptance**, the benchmark defines a technically demanding target for multimodal CAD assistants, tool-using foundation models, and future CAD editing systems [2604.16170].

Source: https://www.emergentmind.com/topics/neuralcad-edit