Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects

Abstract: Articulation modeling aims to infer movable parts and their motion parameters for a 3D object, enabling interactive animation, simulation, and shape editing. In this paper, we present Sketch2Arti, the first sketch-based articulation modeling system for CAD objects. Our key observation is that designers naturally communicate articulation intent through lightweight sketches (e.g., arrows and strokes) that indicate how parts should move, yet translating such sketches into articulated 3D models remains largely manual. Sketch2Arti bridges this gap by enabling users to specify articulation through simple 2D sketches drawn from a chosen viewpoint. Given a CAD model and user sketches, our approach automatically discovers the corresponding movable parts and predicts their motion parameters, allowing iterative modeling of multiple articulations on complex objects with fine-grained control. Importantly, Sketch2Arti is trained in a category-agnostic manner without requiring object category information, leading to strong generalization to diverse objects beyond existing articulation datasets. Moreover, for shell models lacking interior structures, Sketch2Arti supports controllable internal completion guided by user sketches, generating plausible internal components consistent with the existing geometry and predicted motion constraints. Comprehensive experiments and user evaluations demonstrate the effectiveness, controllability, and generalization of Sketch2Arti. The code, dataset, and the prototype system are at https://arlo-yang.github.io/Sketch2Arti.

• The paper presents a novel method to convert designer sketches into accurate kinematic CAD models by leveraging a U-Net and local geometry cues.
• It employs hierarchical segmentation and generative interior completion to refine part localization and motion parameters for improved accuracy.
• Rigorous testing on the new SketchMobility dataset demonstrates superior performance over state-of-the-art methods across in-domain and out-of-distribution scenarios.

Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects

Introduction and Motivation

Articulation modeling is critical for interactive design, simulation, and editing of 3D objects, as it enables the inference of movable parts and their motion parameters. "Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects" (2604.25781) addresses a practical but unstudied need in the CAD and design community: the ability to specify object articulation intent directly through simple user sketches, bypassing the need for exhaustive manual rigging or reliance on limited-category datasets.

The motivation originates from the routine practice among designers who communicate articulation through arrow-like strokes and other geometric indicators during early ideation (Figure 1).

Figure 1: Articulation modeling in the design field: designers sketch arrows and strokes to indicate motion cues, internal structures, and post-articulation geometry during product ideation.

The paper identifies three main challenges for automating the translation of sketches to articulated CAD models: (1) part localization (“where”); (2) articulation parameter estimation (“how”); and (3) handling occluded or missing structural components that become relevant upon articulation (“what”).

System Design: Sketch2Arti

Sketch2Arti formulates sketch-based articulation modeling as a hybrid problem of visual understanding and geometric reasoning. The system is designed for highly controllable and iterative user interaction: a user selects a viewpoint on a 3D model and draws strokes directly to express articulation intent. Rotational motions are indicated by a hinge axis line and directional arrow; translations by a single arrow. The system then translates these cues into an explicit kinematic model.

Key design elements include:

• Category-Agnostic Articulation Analysis: Rather than leveraging category priors, the neural articulation predictor only considers local geometry (depth, normal maps) and the overlaid sketch, enabling generalization beyond fixed benchmarks.
• Part Segmentation via Feature Fields: Sketch2Arti leverages the foundation model Partfield [liu2025partfield] for instance-agnostic segmentation, using hierarchical clustering and the sketch to select the correct articulation part.
• Motion Parameter Post-Processing: Snapping articulation parameters to local geometric anchors (e.g., part boundaries or principal directions), improving physical plausibility and eliminating small but critical misalignments.
• Articulation-Aware Interior Completion: For shell models or missing internal components, the system adapts the 3D generative model Trellis [xiang2025structured] with masked and iterative inpainting, ensuring completed geometry both matches the visible design and satisfies motion constraints.

Dataset: SketchMobility

A new large-scale dataset, SketchMobility, was constructed, containing over 5,000 articulated shapes spanning 48 categories. For each shape, the dataset provides both the 3D model, expert-assigned part and articulation labels, motion parameters, and a suite of synthetically-generated sketch cues emulating realistic designer input. The diversity extends to uncommon object classes such as windmills and motorbikes, which are typically missing from existing benchmarks.

The dataset synthesis pipeline projects 3D motion cues (e.g., hinge vectors, rotational arcs) into image-space, followed by stochastic perturbation at the pixel level to better match freehand sketches.

Quantitative and Qualitative Evaluation

A comprehensive evaluation against state-of-the-art methods including FreeArt3D [chen2025freeart3d] and Singapo [liu2024singapo] demonstrates superior performance on all primary articulation metrics (F-Score, Chamfer Distance, joint axis and pivot error) across both in-domain and out-of-distribution categories. Notably, Sketch2Arti’s articulation prediction is robust to novel categories where baseline approaches fail (e.g., continuous rotation in windmills, vehicle wheels). Representative results are shown in the main gallery figures, while comparative ablation studies justify the selection of hierarchical segmentation over flat k-means and the necessity of post-processing for articulated part motion.

Strong numerical results include: On seven shared benchmark categories, Sketch2Arti achieves a 2.0%/15.3% F-score improvement and substantial reductions in axis/pivot error over FreeArt3D/Singapo. In out-of-domain generalization, crucial motion parameters for complex shapes are correctly inferred where other methods fail.

A formal user study with novice designers confirms the usability and expressiveness of the system—users can quickly author complex articulations, and the system accurately interprets a spectrum of stroke patterns with minimal ambiguity.

Technical Mechanisms

• Articulation Predictor: A U-Net-based multi-head CNN operates on 5-channel image extracts (sketch, depth, normal) from the local region. The predictor outputs a part mask, hinge/pivot location, motion type (classification: translation/rotation), and the motion direction in local coordinates.
• Hierarchical Segmentation: The 2D mask is back-projected onto the mesh; the optimal segment is selected from a hierarchical clustering tree over Partfield features, maximizing alignment with the mask.
• Interior Completion: Generative completion is performed in a flow-matching latent space, with explicit masking to preserve known geometry, iterative refinement to overcome bias toward shell-only training distributions, and strict pruning to maintain motion validity. Decoupled extraction of moving and static parts ensures URDF-compatibility for downstream simulation.
• Failure Mode: The method is currently limited to single degree-of-freedom (DoF) articulation. Mechanisms requiring cascaded or coupled kinematics—e.g., an umbrella’s opening—are outside the method’s current expressive range.

Visualization of Major Components

Figures in the paper effectively elucidate the full pipeline: from a user interface for sketching and selection, through articulation component extraction and articulation parameter estimation, to interior generative completion for articulated states. Results visualizations clarify the system’s capacity for both fine-grained and multi-part modeling.

Implications and Future Directions

This work positions sketch as a first-class modality for functional geometry modeling, expanding the achievable space of articulated asset authoring from a handful of fixed categories to truly open-world, designer-driven applications. The methodology’s agnostic stance toward object category and reliance on local cues is critical for generalization and matches the real needs of engineering, animation, and robotics workflows.

Immediate practical impacts include rapid prototyping of complex mechanisms, designer-in-the-loop authoring, and minimal-bias augmentation of 3D datasets with plausible, functionally aware motion annotations.

Looking forward, major open directions include extending to higher-order or coupled kinematics, tighter mesh-level semantic segmentation for arbitrary tessellation, and unified systems that capture both shape and articulation intent in a single sketch-driven pass.

Conclusion

Sketch2Arti delivers a robust, high-controllability framework for sketch-driven articulation modeling of CAD objects, with rigorous numerical and user-driven validation. Its technical innovations span robust, local sketch-based articulation inference, structure-preserving generative completion, and a high-quality supporting dataset. The approach establishes a framework for interactive, generalized articulation authoring poised to impact physical simulation, animation, and functional asset design.