neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing

Abstract: We introduce neuralCAD-Edit, the first benchmark for editing 3D CAD models collected from expert CAD engineers. Instead of text conditioning as in prior works, we collect realistic CAD editing requests by capturing videos of professional designers, interacting directly with CAD models in CAD software, while talking, pointing and drawing. We recruited ten consenting designers to contribute to this contained study. We benchmark leading foundation models against human CAD experts carrying out edits, and find a large performance gap in both automatic metrics and human evaluations. Even the best foundation model (GPT 5.2) scores 53% lower (absolute) than CAD experts in human acceptance trials, demonstrating the challenge of neuralCAD-Edit. We hope neuralCAD-Edit will provide a solid foundation against which 3D CAD editing approaches and foundation models can be developed. Code/data: https://autodeskailab.github.io/neuralCAD-Edit

• The paper introduces a benchmark for multimodal 3D CAD editing that leverages expert instructions via text, speech, gesture, and drawing.
• The methodology employs automatic metrics (Chamfer, IoU, DINOv2) alongside human ratings to rigorously compare AI outputs with expert edits.
• Results reveal a significant performance gap between AI models and human experts, emphasizing the need for enhanced spatial reasoning and collaborative systems.

neuralCAD-Edit: Multimodal-Instructed 3D CAD Model Editing Benchmark

Introduction

neuralCAD-Edit establishes a comprehensive evaluation framework for editing 3D CAD models driven by realistic, multimodal expert instructions. The benchmark is notable for capturing the natural multimodal behaviors of domain experts—including text, speech, gestural interaction, and drawing—when conveying complex design objectives to others. Unlike prior CAD editing benchmarks that focus exclusively on text interfaces or synthetic data, neuralCAD-Edit collects expert requests as they would occur in professional collaborative environments, leveraging simultaneous manipulation and interaction in production-grade CAD systems. The dataset facilitates direct, quantitative comparison of AI model performance and human expertise in 3D CAD editing under natural, professionally motivated constraints.

Figure 1: Overview of neuralCAD-Edit workflow, including multimodal request capture and benchmark evaluation against both human and AI model edits.

Benchmark Design and Protocol

The dataset centers on a protocol wherein each participating expert designer generates three edit requests (easy, medium, hard) for each input CAD model, reflecting tasks projected to take 2, 5, and 10 minutes, respectively. The requests are delivered through four distinct modalities—text input, interactive (speech and gesture), draw-temp (ephemeral annotations), and draw-static (persistent annotations)—capturing the full spectrum of communication tools available to designers.

For each request, two edits are performed: the initial edit by the requestor and a second independent edit by another expert, both using Autodesk Fusion integrated with a logging and video capture toolchain. Edits are then evaluated using a suite of automatic 3D feature comparison metrics, as well as expert human and VLM-based subjective ratings.

Figure 2: Diversity of request difficulty for a given model, underscoring task complexity range and expert-driven calibrations.

The curated dataset encompasses single- and multi-body models, both parametric and “dumb,” sourced and filtered from the Fusion360 Gallery, ensuring a realistic coverage of model archetypes encountered in professional practice.

Multimodal Request Modalities

neuralCAD-Edit is the first CAD editing dataset to include synchronized, rich multimodal instruction channels, captured directly during interactive design:

Figure 3: Modality examples: text, interactive (with speech and gesture), and drawing-based requests with relevant human edit renders.

• Text: Baseline channel, allowing for isolated analysis of language-driven model limitations.
• Interactive: Screen- and audio-recorded requests, accommodating spoken instructions paired with gestural mouse or viewport manipulations.
• Draw-temp / Draw-static: Requests augmented with sketch-like temporal or persistent overlays, affording designers a nuanced, context-rich method of conveying intent.

Comprehensive logging includes viewport states, execution events, and timestamped automatic and oracle-verified transcripts. The data enables fine-grained analysis of context alignment between verbal, gestural, and spatial cues.

Dataset Analysis

The dataset comprises 192 requests (1.9 hours of requests) and 384 edits (28.4 hours editing time) from ten experts with 8–13 years of design experience. Each parametricity/construction/modality/difficulty axis is systematically covered to maximize representational diversity. Analysis of request properties reveals that multimodal (speech and sketch) requests consistently convey more information content per unit time compared to typing, and support more complex downstream edits.

Figure 4: Distribution of request lengths across modalities and difficulties, highlighting modality-driven expressivity.

Figure 5: Efficiency scatter: Time versus human acceptance score for edits by modality and agent.

Edits corresponding to drawing-augmented requests required more user actions and time, especially for harder tasks, indicating that multimodal instructions facilitate complex modifications that go beyond the expressive limits of text-only interfaces.

Experimental Evaluation

Baseline and Model Harness

The benchmark employs a rigorous experimental setup:

• Human Upper Bound: The requestor edit and a “human baseline” (non-requestor CAD expert), capturing expert execution variability.
• AI Systems: Google Gemini 3 Pro (native video/speech), GPT 5.2 (multiframe image sampling from video), and Claude Sonnet 4.5; all operating within the same CadQuery-based tool interface allowing multi-step scripting, conditional inspection, and iteration until task self-termination.
• Automatic Metrics: Chamfer distance, Voxel IoU, DINOv2 similarity, and edit validity.
• Subjective Metrics: Blind evaluation by five expert raters on instruction-following and model quality using strict rubrics; VLM-based rating (primarily GPT 5.2).

Main Results

All frontier AI models exhibit a substantial performance deficit compared to human baselines. The highest-performing model (GPT 5.2) achieved acceptance in only 25% of cases (absolute 53% below human performance), while Gemini and Claude fared worse. Notably, human inter-rater consensus was high (ICC(2,k) = 0.88), indicating stable ground-truth calibration and a robust differentiation between model outputs.

Figure 6: Qualitative comparison: AI edits versus human standard, revealing frequent AI failures in compositional 3D reasoning (misplaced features, misconstruction) even when initial intent is partially matched.

No AI model demonstrates reliable improvement in edit quality or time spent as task difficulty increases, in stark contrast to expert humans, who allocate more time and deliver higher-complexity edits on hard tasks. This points to persistent limitations in AI agent sensitivity to real-world task complexity and the lack of adaptive effort allocation—essential for practical workflow integration.

Figure 7: Modality analysis: Human editors exploit extra information in multimodal requests, achieving highest acceptance when drawing is involved; current AIs gain little or no benefit.

Metrics and Evaluation Methodology

Correlation analysis demonstrates that VLM-based evaluations, while directionally consistent with human scorer rankings, overrate model outputs and underestimate human high-quality edits. Automatic feature-based metrics, particularly DINOv2, offer moderate correlation with subjective acceptance but fail on semantically ambiguous, visually similar cases, underscoring the need for robust human-in-the-loop validation in future cycles.

Figure 8: Spearman correlations between human, VLM, and automatic metrics: No purely automated approach adequately substitutes for expert human assessment.

Examining “cross-evaluator” settings reveals that VLMs self-assess generously and converge poorly on the true human performance margin, emphasizing that VLM-based metrics are inadequate for progress reporting and discrimination as AI edit fidelity improves.

Implications and Future Directions

This work defines a state-of-the-art challenge for evaluating multimodal-instructed 3D geometric editing—setting a high bar for foundation model adoption within professional CAD and design workflows. The strong separation between expert human and machine performance (even under unconstrained, tool-use-enabled “agent” settings) demonstrates persistent deficiencies in 3D spatial reasoning, long-horizon geometric planning, and compositional visual-linguistic understanding in leading models.

Theoretical implications include the necessity of joint modeling for vision, language, spatio-temporal manipulation, and tool-use planning to capture the full breadth of designer intent and execute complex edits that align with professional standards. Practically, the poor performance at the current frontier suggests that human-in-the-loop, collaborative systems—and training paradigms leveraging expert-in-the-loop feedback on multimodal datasets like neuralCAD-Edit—will remain critical for the foreseeable future.

Future research should focus on large-scale multimodal pretraining on expert-annotated interactive CAD data, improving model architectures for compositional and spatial reasoning, adaptive effort allocation, and harnessing collaborative human–AI approaches for iterative design refinement.

Conclusion

neuralCAD-Edit positions itself as a foundational resource for measuring progress in multimodal 3D CAD editing. By capturing natural expert communication modalities and providing rigorous baselines, it exposes acute gaps between AI capabilities and professional geometric design standards. The benchmark will catalyze method development across 3D understanding, multimodal instruction following, and agentic tool-use, driving the next wave of AI advances in computer-aided design.