Sequence-to-Sequence Learning for CAD

• Sequence-to-sequence learning for CAD is a paradigm that transforms diverse inputs into a series of parametric design operations using neural networks.
• It leverages varied representations such as parametric command tuples, DSL trees, and programmatic code to ensure geometry fidelity and editable construction histories.
• Advanced architectures like Transformers and memory-augmented models enable accurate command prediction and integrate multi-modal data for enhanced design automation.

Sequence-to-sequence learning for Computer-Aided Design (CAD) refers to the paradigm in which a neural architecture transforms an input sequence—such as a textual description, drawing, image, point cloud, or other structured/informal design specification—into an output sequence representing stepwise CAD operations (e.g., sketch commands, construction parameters, or executable procedures). This formulation brings advances from generative modeling, memory-augmented computation, and deep sequential transformation to address the complexity, hierarchy, and expressivity requirements of modern CAD workflows.

1. Foundations and Sequence Representations

Sequence-to-sequence CAD models encode both the structural and geometric information intrinsic to CAD designs. An input (e.g., a vectorized drawing, text prompt, image, or point cloud) is transformed, through a learnable mapping, into an ordered series of CAD operations that can reconstruct the target model in a parametric, programmatic, or executable form.

Key sequence representations in CAD applications include:

• Parametric command tuples: Each element is represented as $(T, p_1, ..., p_n)$ where $T$ is the command type (e.g., "line," "arc," "extrude") and $p_i$ are geometric or topological parameters (Qin et al., 26 Aug 2025).
• Domain-specific language (DSL) trees or feature sequences: Operations mapped to intermediary or vendor-specific scripting languages, including both symbolic and numerical attributes (Li et al., 9 Jan 2025).
• Programmatic code: Entire design histories rendered as direct Python code using CAD libraries like CadQuery, supporting explicit procedural editing and semantic introspection (Rukhovich et al., 18 Dec 2024).
• JSON-based hierarchical sequences: Human- and machine-readable structures representing sketches, primitives, and operations in a minimal, lossless schema (Govindarajan et al., 13 Jul 2025).

The output sequence typically preserves the construction history, enabling both geometry retrieval and parametric editability.

2. Model Architectures and Learning Strategies

Sequence-to-sequence CAD learning employs a diverse spectrum of architectures, all centered around a mapping $f(x) \mapsto y_1, y_2, ..., y_N$:

a. Memory-Augmented and Deep Hierarchical Models

• DeepMemory (Meng et al., 2015): Stacks multiple layers of memory modules, each layer effecting a nonlinear transformation via read–write operations with content- or location-based addressing. Controllers (RNN, LSTM) update read states via content-based attention, enabling the network to focus dynamically on relevant parts of the design specification and propagate long-range dependencies.

b. Transformer-Based Architectures

• Dual-Decoder Transformer: As in Drawing2CAD, a shared encoder extracts features from vectorized drawing primitives, feeding into a command-type decoder (classification over $T$) and a parameter decoder (continuous regression over $p_i$) (Qin et al., 26 Aug 2025).
• Hierarchical Transformers: Used for processing designs exhibiting sketch–extrude hierarchies, employing modular decoders for loop primitive generation and refinement at multiple abstraction levels (Dupont et al., 17 Jul 2024).
• Contrastive and Autoencoding Models: Encoder–decoders with contrastive latent spaces (e.g., ContrastCAD), where representations are regularized such that CAD models with similar topology shape are embedded nearby, even under sequence permutation (Jung et al., 2 Apr 2024).

c. Distribution Matching and Augmentation

• Distribution matching frameworks approximate local latent distributions around each design instance, employing RNN-based augmenters to generate diverse synthetic sequence variants; learning minimizes KL divergence between inferred source and target distributions (Chen et al., 2018).

d. Multimodal and Code-LLMs

• Generative models interfacing text, drawing, image, and code modalities, including LLMs fine-tuned for direct sequence-to-sequence mapping in programmatic or minimally annotated formats (Govindarajan et al., 13 Jul 2025, Rukhovich et al., 18 Dec 2024).

3. Sequence Transformation Operations and Losses

Sequence transformation in CAD demands both fidelity in command type prediction and geometric precision in continuous parameters. Network training commonly employs:

• Soft target distribution loss: Rather than one-hot target parameterization, supervision is provided by soft distributions over plausible parameter values when ambiguity exists, increasing tolerance to specification variability (Qin et al., 26 Aug 2025).
• Composite losses: Hierarchically sum classification loss (for $T$) and mean squared error or negative log-likelihood over parameters ($p_i$), sometimes regularized for sequence length or diversity (Dupont et al., 17 Jul 2024).
• Augmented KL divergence: Incorporates entropy regularization for augmented sequence generation and fidelity to original design prototypes (Chen et al., 2018).

4. Data, Evaluation Metrics, and Experimental Protocols

Performance evaluation for sequence-to-sequence CAD synthesis leverages both standard and specialized metrics:

Metric	Domain Evaluated	Interpretation/Usage
Command Accuracy (ACC)	Sequence/Operation Level	Fraction of correctly predicted command types/placeholders
Parametric MSE/CD/etc.	Geometry	Distance between predicted and ground-truth geometries
F1 score	Operation/Primitive	Hungarian-matched accuracy of lines, arcs, circles, extrude ops
Sequence Similarity	Sequence	Levenshtein (edit) distance, CAD Sequence Similarity Score (CSSS)
Mean Average Precision	Sequence + Parameter	Evaluates correctness of both type and continuous parameters
Structural Topology	Mesh/Topology	Sphericity discrepancy, mean curvature, Euler characteristic, etc.
Invalidity Ratio (IR)	Validity	Percentage of syntactically/geometrically invalid sequences
IoU	Geometry	Intersection-over-union of generated and reference 3D CAD models

Data for training and benchmark comes from paired vector drawing–CAD model datasets (CAD-VGDrawing (Qin et al., 26 Aug 2025)), synthetic program-image pairs (Li et al., 9 Jan 2025), expansive annotation campaigns using GPT-4.1 (Govindarajan et al., 13 Jul 2025), or large procedurally-generated code corpora (Rukhovich et al., 18 Dec 2024), supporting both model generalization and detailed structural evaluation.

5. Applications and Workflow Integration

Sequence-to-sequence techniques for CAD have been successfully applied to diverse workflows:

• Engineering Drawing Conversion: Automatic parameterization and reconstruction from 2D vectorized or hand-drawn sketches (Drawing2CAD (Qin et al., 26 Aug 2025), Sketch2CAD (Li et al., 2020)).
• Reverse Engineering: Inference of editable, interpretable CAD code or DSL from point clouds or images, enabling legacy part recovery, inspection, or medical reconstruction (Rukhovich et al., 18 Dec 2024, Dupont et al., 17 Jul 2024, Li et al., 9 Jan 2025).
• AI-Driven Design Automation: Text- or image-driven parametric modeling, where designer-level commands or product photos serve as input to generate full construction histories directly (Govindarajan et al., 13 Jul 2025, Khan et al., 25 Sep 2024, 2505.19490).
• Interactive and Flexible Editing: Outputting programmatic CAD (e.g., CadQuery Python scripts) allows direct code-level modification, question answering, or natural language post-editing via LLMs (Rukhovich et al., 18 Dec 2024).
• Automated Industrial Workflows: Accurate translation of user requirements and geometric constraints into manufacturable CAD sequences, enhancing design reproducibility and data reusability (2505.19490).

6. Baseline Comparisons, Limitations, and Open Challenges

Sequence-to-sequence CAD models generally outperform rule-based, CRF, or standard LSTM/BiLSTM baselines on both sequence and geometric metrics (Guo et al., 2018, Qin et al., 26 Aug 2025, 2505.19490). However, several challenges remain:

• Parameter Ambiguity: A single design intent may correspond to multiple plausible parameterizations; soft target losses and permutation-invariant architectures (contrastive learning) mitigate, but not eliminate, this ambiguity (Jung et al., 2 Apr 2024, Qin et al., 26 Aug 2025).
• Domain Gap: Models trained on synthetic or limited-shape datasets struggle to generalize to highly complex, real-world parts or freeform geometries; overcoming the domain gap via larger, richer datasets and domain adaptation remains a primary research direction (Li et al., 9 Jan 2025).
• Sequence Validity: Ensuring syntactic and geometric correctness—especially for longer, multi-stage sequences—remains challenging. Methods incorporating validity checks, code-execution, or spatial reasoning mechanisms (as in CAD-GPT) consistently achieve lower invalidity ratios (Wang et al., 27 Dec 2024).
• Hierarchical and Multi-modal Complexity: CAD workflows often feature deeply nested or nonlocal dependencies, requiring models to integrate both hierarchical structure and multimodal cues (text, image, drawing, point cloud); continued architectural research is needed to scale sequence-to-sequence learning to these settings.

7. Future Directions

Key research frontiers include:

• Augmented and Multi-modal Learning: Joint reasoning across text, code, image, and 2D/3D geometry to handle richer design scenarios and natural human–machine interaction contexts.
• Hierarchical and Modular Representation: Accommodating assemblies, feature groups, and advanced parametric dependencies through nested sequence modeling and refined hierarchical decoders (Dupont et al., 17 Jul 2024).
• Interactive and Post-hoc Editing: Enabling iterative editing, correction, and guided modification of sequences using semantic-level control signals and LLM post-processing (Rukhovich et al., 18 Dec 2024).
• Uncertainty Quantification: Explicitly modeling uncertainty in parameter prediction and structure generation to support design space exploration and robust downstream applications (Qin et al., 26 Aug 2025).
• Verification and Feedback Loops: Integration with CAD kernels for real-time validity checks and optimization-refined sequence prediction in-the-loop (Wang et al., 27 Dec 2024, Yin et al., 24 Mar 2025).

Sequence-to-sequence learning for CAD thus stands as a convergence point for deep sequence models, geometric reasoning, and industrial design automation—offering a unified, flexible, and scalable formulation for transforming diverse input modalities into structured, editable CAD construction sequences.