Sketch2CAD: From Sketch to Editable CAD

• The system converts sequential 2D sketches into fully editable CAD models using deep neural networks for operator prediction and context-aware segmentation.
• It integrates geometric optimization techniques such as curve fitting and line search to extract precise parameters from freehand inputs.
• The framework supports diverse CAD operations with a full edit history, enabling rapid prototyping via an intuitive, context-driven interface.

A Sketch2CAD system is an interactive computational framework that translates sequential 2D user sketches into parametric, editable CAD models by interpreting hand-drawn inputs in the geometric context of the evolving design. It unifies advances in deep learning, computer-aided design representations, and human–computer interaction to enable efficient model authoring and rapid prototyping through sketch-based workflows.

1. System Architecture and Workflow

The Sketch2CAD system operates as an incremental modeling platform wherein users construct complex 3D CAD objects by successively drawing overviews or edits directly on the current model state. At each step, the system captures the latest 2D sketch—typically a set of freehand strokes drawn atop a rendered image of a partial CAD model—alongside contextual geometric cues such as depth and normal maps of the existing shape. This input tuple is processed by a deep neural pipeline to infer both the intended CAD operation (e.g., extrude, bevel, boolean add/subtract, or sweep) and the associated parameter set required to unambiguously enact that operation on the current model.

After operator and parameter inference, an optimization routine (e.g., line search, curve fitting) computes precise geometric values, which are then used to update the CAD model. The system maintains a protocol—a complete edit history—capturing each step as an explicit operator plus its parameterization, ensuring full editability and supporting downstream modifications or re-use (Li et al., 2020).

2. Neural Network Design and Contextual Parsing

Central to disambiguating freehand input is the coupling of convolutional or encoder–decoder (U-Net) architectures for both classification and image segmentation. The classifier consumes the 2D sketch and context (256×256 images of sketch, depth, and normal channels), outputting the predicted CAD operator type. Subsequently, one or more segmentation networks partition the input image into operational subregions—such as base faces, offset strokes, or profile curves—critical for recovering operation parameters.

Notably, the system leverages context (the geometric state of the partial model) at every step, addressing a longstanding challenge of sketch-based modeling: the inherent ambiguity in 2D stroke interpretation. Rather than regressing geometric parameters directly—which is unstable—the segmentation networks output spatial masks, which are post-processed using geometric techniques:

• Stroke pixel counting for discrete length estimation
• Curve fitting for profile extraction
• Projection and distance minimization (e.g., for offset calculations):

$$\mathrm{dist}(d) = \sum_{p} \min_{q \in C_o} \left\| \pi_v(p + d \cdot n_f) - q \right\|$$

where $p$ are points on a planar face boundary, $n_f$ is the face normal, $\pi_v$ denotes projection, and $C_o$ is the curve mask (Li et al., 2020).

3. Supported CAD Operations and Parameterization

The system supports four canonical operations:

• Extrude: Displaces a planar face by an offset $d$ along the surface normal.
• Bevel (Rounding): Modifies an edge/corner by replacing it with a profile curve, parameterized by two parallel profile strokes and a direction vector.
• Add/Subtract: Inserts or removes a prism (with up to 6-edge bases) on a face, parameterized by a base polygon, profile length, and a union/difference Boolean flag. The network regresses ordered base/offset curves to resolve topological ambiguity.
• Sweep: Constructs a cylindrical or ellipsoidal protrusion by specifying paired circular/elliptical cross sections, endpoints, and a profile curve.

Each operation’s parametrization is deduced from semantic sketch segmentation and geometric optimization, ensuring an explicit mapping from 2D intent to procedural CAD commands.

4. Synthetic Data Generation and Network Training

In the absence of real paired sketch–CAD sequences, training relies on synthetic data. Randomized multi-step operator sequences generate diverse CAD models, which are rendered into line drawings (with stochastic perturbations for “hand-drawn” realism) alongside depth and normal maps. Balanced operator frequencies and multiple sequence lengths ensure coverage of plausible design workflows (e.g., 40,000 shapes across four operations).

Supervised training uses weighted cross-entropy for operator classification and $L_2$ regression for segmentation masks. This approach enables generalization over execution order and stylistic stroke variation observed in practical sketching (Li et al., 2020).

5. User Interface, Interaction Features, and Performance

The user interface provides real-time visualization of the evolving 3D model, allowing sketching directly onto rendered model views. Integrated stroke regularization (including snapping, rectification, and symmetry-based auto-completion) enforces geometric precision. Parameter tuning widgets offer manual override of recovered values, enabling users to resolve network ambiguity or specify exact tolerances. Edit histories are persistent and revisable.

Empirically, network inference is rapid (≈0.07 s per step); geometric search/optimization phases range from 0.01–2 s, and Boolean drafting is handled at ≈0.02 s per operation on commodity GPUs. User studies confirm that novices rapidly complete modeling tasks (5–20 min per shape), highlighting usability and translation accuracy.

Module	Time per operation (s)	Function
Classifier/UNet	0.07	Operator/Segmentation
Geometric Search	0.01–2	Offset/Curve Fitting
Boolean Ops	0.02	Add/Subtract/Sweep

6. Addressing Ambiguity and Generalization

Ambiguity in sketch interpretation—stemming from overlapping strokes and operator similarity—is mitigated by mandatory context inputs (depth/normal of the current model). The two-stage (classification + segmentation) neural pipeline, with robust intermediate representations, sidesteps the instability of direct parameter regression. Weighted loss functions and synthetically diversified training data further increase resilience to freehand noise and stroke variability.

The architecture is currently restricted to operations on planar faces. Increasing representation power (e.g., to sketches on curved surfaces or operations involving non-Euclidean parameterizations such as NURBS) is identified as a critical extension.

7. Future Directions and Limitations

Future directions include:

• Curved Surface Editing: Extending support to NURBS or spline-based surfaces.
• Semantic and Intent Modeling: Leveraging annotated datasets (e.g., PartNet) or real user edit histories to infuse domain semantics.
• Multi-Modal Interfaces: Adapting mouse-based inputs to stylus/tablet modalities for enhanced expressiveness.
• Bidirectional Interaction: Enabling “re-sketching” of CAD models using NPR to close the loop between ideation and procedural modeling.

A current limitation is the lack of explicit part semantics or design intent in synthesized training data. Incorporating semantic part annotation and typical industrial operation protocols would further enhance the system’s practical relevance.

---

Sketch2CAD demonstrates that a fusion of deep contextual parsing, synthetic supervision, and formalized operation parameterization achieves robust, intuitive, and precise translation of 2D freehand sketches to fully procedural, editable CAD models (Li et al., 2020). The workflow replicates the sequential, stepwise reasoning of industrial designers while providing immediate, precise digital feedback and editability appropriate for design iteration, manufacturing, and procedural design reuse.