ControlSketch-Part: Incremental Vector Sketching
- ControlSketch-Part is a part-annotated vector sketch dataset enabling semantic decomposition and incremental sketch generation.
- It employs a multi-stage automatic annotation pipeline using VLMs to generate global captions, part descriptions, and precise SVG path assignments.
- The training regime combines supervised fine-tuning and process-reward reinforcement learning to enhance interpretability, controllability, and local editability.
ControlSketch-Part is a part-annotated vector-sketch dataset and training substrate introduced to support text-to-vector sketch generation “one semantic part at a time” with visual feedback (Du et al., 19 Mar 2026). It extends the ControlSketch dataset, which contains 35,000 image-sketch pairs for 15 object categories, by replacing whole-sketch supervision with structured part-level supervision: a short global caption, a set of semantic part descriptions, and a complete path-to-part assignment in which every SVG path is assigned to exactly one part. The associated generation framework uses this structure to train a multi-modal LLM-based agent that draws incrementally rather than producing a sketch in a single pass, with the stated goals of interpretability, controllability, and local editability.
1. Definition and problem setting
ControlSketch-Part was created to address a specific limitation of prior text-to-vector sketch systems: they typically generate an entire sketch in one pass, which is difficult to edit, hard to interpret, and poorly aligned with how humans sketch (Du et al., 19 Mar 2026). The dataset operationalizes an alternative view of sketch generation in which the model draws one part, visually checks the evolving canvas, and then draws the next part. In this formulation, the supervision is semantic and sequential rather than only global.
The paper defines the essential supervision as three aligned layers. First, each sketch has a brief literal description of the whole object. Second, it has semantic descriptions of its components, expressed as object parts rather than drawing primitives. Third, it has a path-level correspondence so that every vector path belongs to one semantic part. The number of parts per sketch is constrained to be between 2 and 5 (Du et al., 19 Mar 2026).
A common misconception is to treat ControlSketch-Part as a generic vector-sketch corpus or as a conventional segmentation benchmark. The paper instead frames it as a dataset for semantic decomposition and incremental generation. The segmentation is “semantic decomposition,” not geometric segmentation in the usual computer vision sense, and the path assignment is intended to expose supervision at the vector-primitive level rather than merely labeling an entire sketch (Du et al., 19 Mar 2026).
2. Dataset composition and annotation semantics
ControlSketch-Part is built from the ControlSketch dataset. The original sketches are described as professional-quality vector sketches generated by an SDXL + SDS-based optimization pipeline, and the paper states that category labels are not used as training supervision; they are used only for organizing examples in the paper (Du et al., 19 Mar 2026). This design choice matters because the target task is conditioned on language and canvas state, not on a closed-set category classifier.
The dataset enriches each sketch with three annotation types:
| Annotation type | Content |
|---|---|
| Global caption | A short literal description of the whole sketch |
| Part descriptions | Semantic descriptions of the sketch’s components |
| Path-to-part assignment | A complete mapping from every SVG path to exactly one part |
The part descriptions are intended to be visually grounded, mutually non-overlapping, collectively exhaustive, concrete, and specific (Du et al., 19 Mar 2026). The instructions explicitly exclude high-level object labels such as “a dog” or “a woman,” drawing-language descriptions such as “strokes” or “lines,” and style, color, or composition language. This makes the representation suitable for a policy that conditions on meaningful object components rather than on low-level mark-making categories.
The path-to-part assignment is equally central. After parts are defined, each SVG path is mapped to exactly one semantic part, every path must be assigned, and every part must be used at least once (Du et al., 19 Mar 2026). This transforms the dataset from a captioned sketch collection into a part-supervised vector corpus capable of supporting turn-by-turn generation and localized modification.
3. Automatic annotation pipeline
A principal contribution of the work is a scalable automatic pipeline that uses a VLM to annotate parts in raw SVG sketches (Du et al., 19 Mar 2026). The pipeline is described as generic and intended to be applicable to any vector sketch dataset. It is deliberately multi-stage and includes critique-and-revision loops, reflecting the paper’s observation that even strong VLMs are imperfect at complex structured annotation tasks.
The first stage is initial part decomposition. The SVG sketch is rendered to an image, and a VLM proposes a small set of semantic parts that jointly and exhaustively describe the object. The output is a JSON array of strings, each string being one part description (Du et al., 19 Mar 2026). A second VLM pass critiques this proposal against the original instructions and returns a structured list of issues, a summary, and a boolean should_revise. Each issue includes type, severity, reason, and suggested_fix. If needed, a refinement step rewrites the part decomposition while preserving the JSON array format.
The next stages operate at the vector-path level. Using the refined parts, the SVG source, and the rendered sketch, the pipeline performs an initial path assignment whose output is a JSON object with keys Path1, Path2, ... and values Part1, Part2, ... (Du et al., 19 Mar 2026). This assignment is then audited in a stronger critique step aided by a diagnostic visualization: the left panel shows each part label and its description in a unique color, while the right panel recolors the sketch paths using the same part colors. If mismatches are detected, the path assignment is revised under the same schema constraints. Finally, the VLM generates a short global caption using only the refined parts, which is intended to keep the overall caption consistent with the part-level structure (Du et al., 19 Mar 2026).
This pipeline establishes the dataset’s defining property: a structured alignment between rendered appearance, semantic parts, and SVG paths. A plausible implication is that the critique-and-refinement design is as important as the initial proposal stage, because the supervision target is not merely descriptive text but a constrained, machine-usable decomposition.
4. Training regime and process-reward reinforcement learning
The training framework built on ControlSketch-Part has two stages: supervised fine-tuning and reinforcement learning with multi-turn process-reward GRPO (Du et al., 19 Mar 2026). The supervised stage teaches the model the basic output format and a single-turn sketching policy. At each training example, the model receives the rendering of the current canvas, a short global caption, the description of the next part to draw, descriptions and vector paths of already drawn parts, and the number of parts remaining. Its output is the vector path or paths for the next part, encoded as cubic Bézier commands only.
To avoid fixing a canonical drawing order, the paper uses permutation-based training. Each sketch is augmented by sampling up to 20 permutations of its parts, so that the same final sketch yields multiple progressive drawing sequences (Du et al., 19 Mar 2026). The resulting training pairs simulate a sequence such as empty canvas plus part description, then partial canvas plus next-part description, and so on. The loss is standard cross-entropy or next-token prediction.
The second stage addresses a train-test mismatch. During supervised fine-tuning, the model mainly sees oracle intermediate canvases from ground truth; at inference time it must condition on its own previous, imperfect outputs (Du et al., 19 Mar 2026). The paper therefore treats sketching as a sequence of turns and assigns rewards at intermediate steps rather than only at the end. In standard GRPO, cumulative normalized rewards define the advantage:
In the process-reward variant, because each sketch has a fixed number of parts and each intermediate state can be compared with a corresponding ground-truth intermediate rendering, rewards are normalized within each step:
The reward signal combines DreamSim similarity at the current step with a path-count reward that discourages excessive path generation, using
with in experiments (Du et al., 19 Mar 2026). If an output fails format validation, it receives the minimum reward and its trajectory is terminated. The GRPO objective is a clipped policy-gradient objective with KL regularization. The paper argues that this multi-turn, intermediate-reward formulation closes the gap between oracle intermediate states and self-generated intermediate states, thereby improving progressive sketch quality.
5. Interpretability, controllability, and local editability
The paper identifies three properties enabled by the combination of part-level supervision and visual feedback: interpretability, controllability, and local editability (Du et al., 19 Mar 2026). Interpretability follows from the fact that each turn corresponds to a meaningful semantic part rather than an opaque continuation of the entire sketch. The model’s action sequence can therefore be inspected in semantic units.
Controllability follows from conditioning each turn on the next part description and the current canvas. The user can specify or alter which part should be drawn next, and the evolving canvas changes the subsequent generation context (Du et al., 19 Mar 2026). The examples in the paper reportedly show that changing an early part alters later output in localized ways even when later descriptions remain unchanged. This suggests a control mechanism that is both compositional and state-dependent.
Local editability arises because parts are represented separately and because every SVG path is tied to a semantic part. The paper explicitly highlights localized editing via arbitrary stroke removal and replacement (Du et al., 19 Mar 2026). In contrast to one-shot generation, where editing often requires regenerating the whole sketch, ControlSketch-Part supports workflows in which one component can be removed or replaced without necessarily redrawing unrelated components.
A second misconception is to equate this local editability with unrestricted interactive authoring. The paper does not claim full self-critique during generation, and it notes that errors can still accumulate over many turns (Du et al., 19 Mar 2026). The control is therefore structured and localized, but not unlimited.
6. Evaluation, limitations, and position within related research
The empirical evaluation compares the full system against SketchAgent, Gemini 3.1 Pro, SDXL + SwiftSketch, an SFT-only ablation, single-turn RL, multi-turn outcome-reward RL, and the proposed multi-turn process-reward RL (Du et al., 19 Mar 2026). The automatic metric is Long-CLIP cosine similarity between the final sketch and the concatenated part descriptions. According to the paper, the full model achieves the best score among all methods; ground-truth sketches define an upper-bound reference and random sketches define a lower bound. In ablation, the reported Long-CLIP scores are 0.281 for single-turn RL, 0.286 for multi-turn outcome-reward, and 0.298 for multi-turn process-reward, supporting the claims that multi-turn training outperforms single-turn RL and that intermediate process rewards outperform final-only outcome rewards (Du et al., 19 Mar 2026).
The user studies are double-blind forced-choice studies on Prolific, one for final output quality and one for step-by-step generation quality and part alignment. The paper reports a total of 3,092 pairwise comparisons and states that participants consistently prefer the final model over all baselines (Du et al., 19 Mar 2026). Qualitatively, the full method is described as producing smoother vector paths, more natural overall style, clearer semantic parts, and better structural plausibility. Baseline failure modes are also enumerated: SketchAgent often yields simple, geometric, icon-like forms and sometimes misplaces parts; Gemini 3.1 Pro tends toward simple symmetric structures and can omit details or fail to complete an object; SDXL + SwiftSketch can produce smooth sketches but struggles with long prompts and loses compositional detail (Du et al., 19 Mar 2026).
The limitations are explicit. Failure modes include premature stopping due to the path-count reward, failure on unfamiliar topologies such as a vertically oriented oval rear wheel, and residual misalignment such as a jacket positioned too far from the body (Du et al., 19 Mar 2026). More generally, data coverage is limited to the object types present in ControlSketch; the agent does not yet self-critique during generation; and errors can accumulate over many turns. Suggested future work includes using a planning agent to coordinate multiple agents in parallel, adding self-refinement of intermediate outputs, incorporating more chain-of-thought style reasoning before each part, and extending the agent to other visual reasoning tasks (Du et al., 19 Mar 2026).
Within the broader sketch-control literature, ControlSketch-Part occupies a distinct niche. “Sketch-to-Design: Context-based Part Assembly” formulates interactive sketch-driven 3D design as retrieval, placement, snapping, and stitching of semantically segmented parts from a database, showing an earlier form of part-level control rooted in contextual assembly (Xie et al., 2012). “Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects” likewise adopts a part-aware divide-and-combine strategy, decomposing sketches into semantic components, beautifying individual parts, and then reassembling them through a structure beautification module (Yu et al., 2023). By contrast, “Creating Interactive Behaviors in Early Sketch by Recording and Remixing Crowd Demonstrations” addresses interactive behavior authoring in early GUI sketches through a demonstrate-and-remix workflow rather than vector sketch generation (Lee et al., 2016), while “Sketch Bug: Using Sketch-Based Input for Interactive Code Debugging” uses sketch-like pen input for breakpoint placement and execution control in debugging (Chen et al., 22 May 2026). This comparison suggests that ControlSketch-Part is best understood not as a generic “sketch control” interface, but as a part-annotated supervision framework for incremental, language-conditioned vector sketch synthesis.
