GroupSketch: Sketch Grouping & Animation

• GroupSketch is a dual-purpose framework that groups free-hand sketches into semantically meaningful parts and enables vector sketch animation.
• The original system uses a sequence-to-sequence VAE with both generative and discriminative branches to predict pairwise stroke affinities and coherent grouping.
• The later version leverages interactive semantic grouping and a Group-based Displacement Network to refine multi-object vector sketch animations.

GroupSketch denotes two distinct research systems in the arXiv literature. In its 2018 usage, GroupSketch is a universal sketch perceptual grouper: a model that takes a free-hand sketch of any object category and groups its constituent strokes or stroke segments into semantically meaningful parts without requiring part labels or category-specific training, producing a partition of segments rather than semantic part names (Li et al., 2018). In a later 2025 usage, the name GroupSketch refers to a two-stage, grouping-aware framework for vector sketch animation from text prompts, designed for multi-object scenes and complex motions through interactive semantic grouping, keyframe-based motion initialization, and group-conditioned motion refinement (Liang et al., 21 Aug 2025). Across both usages, the central notion is that grouping provides an intermediate structure between raw vector strokes and higher-level sketch understanding or generation.

1. Universal perceptual grouping as the original meaning

In the 2018 formulation, GroupSketch was introduced to develop a universal sketch grouper that can be applied to sketches of any category and any domain in order to group constituent strokes or segments into semantically meaningful object parts (Li et al., 2018). The target is not category-specific segmentation and not semantic part labeling. Instead, the model infers whether pairs of segments belong to the same part, thereby operationalizing universal perceptual grouping in a category-agnostic manner.

The grouping problem is defined over a vector sketch decomposed into segments $S_i$. For each pair $(S_i, S_j)$, the task is to decide whether the two segments belong to the same part. This is encoded by an affinity matrix $\mathbf{G} \in \mathbb{R}^{N \times N}$, with

$$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$

The implied parts are the connected components induced by affinity $1$, so the output is a partition of the sketch into groups that correspond to object parts meaningful to human annotators (Li et al., 2018).

The motivation for universality is explicitly tied to the difficulty of transferring grouping across diverse shapes, topologies, datasets, and drawing styles. The formulation is challenging because it is category agnostic, because Gestalt cues such as proximity, similarity, continuity, closure, and symmetry are implicit and instance-dependent, because free-hand sketches are sparse and highly variable, and because the model is expected to generalize beyond QuickDraw sketches to edge maps extracted from photos (Li et al., 2018). The same formulation also establishes why grouping is useful: it supports category-agnostic sketch understanding, editing, abstraction, synthesis, and retrieval, and it provides a data-driven way to capture Gestalt grouping for sketches.

2. Dataset foundation: the Sketch Perceptual Grouping dataset

The original GroupSketch system is built around the Sketch Perceptual Grouping (SPG) dataset, described as the largest free-hand sketch perceptual grouping dataset at the time, with 20,000 sketches from 25 object categories, evenly distributed at 800 sketches per category (Li et al., 2018). Categories were selected from QuickDraw under explicit constraints: each had to have at least three semantic parts, and the set was chosen for variety so as to test generalization.

The seen/unseen protocol is central to the claim of universality. Of the 25 categories, 20 categories are used as seen categories for training, validation, and testing, while 5 categories are reserved as unseen categories for testing only (Li et al., 2018). The annotation protocol is vectorial and stroke based: annotators group segments into semantic parts, assign shared group IDs to segments in the same part, and thereby define the ground-truth affinity matrix $\mathbf{G}$.

Several design choices in SPG are explicitly tied to universal grouping. The dataset is balanced across categories, includes diverse semantic types such as animals, vehicles, household objects, tools, and natural objects, and is stored in vector format so that stroke trajectories are preserved for sequential and stroke-based modeling (Li et al., 2018). Quality control includes filtering low-quality sketches with a trained sketch classifier, removing over-abstract sketches with fewer than three parts, removing ambiguous sketches with extraneous background or multiple objects, and keeping only the majority subcategory when necessary.

A concise summary of the dataset structure is given below.

Property	Value
Total sketches	20,000
Categories	25
Sketches per category	800
Seen categories	20
Unseen categories	5
Annotators	25

The annotation policy is also methodologically important. Each annotator was assigned to a single category to ensure intra-category consistency, and the resulting labels provide both grouping assignments and semantic group IDs, although the 2018 GroupSketch model itself uses only the grouping relationships encoded as same-group indicators (Li et al., 2018).

3. Model architecture and grouping objective

The 2018 GroupSketch model uses a sequence-to-sequence Variational Autoencoder (VAE) on vector strokes, extending the Sketch-RNN paradigm with an explicit discriminative grouping branch (Li et al., 2018). Sketches are represented as sequences of stroke segments,

$[S_1, S_2, \ldots, S_N],$

with each segment written as

$S_i = (\Delta x, \Delta y, p),$

where $\Delta x$ and $\Delta y$ are offsets from the previous point and $(S_i, S_j)$0 is the drawing state. Preprocessing includes stroke normalization and data augmentation through stroke removal and distortion.

The encoder is a bidirectional RNN that maps the input sequence to a Gaussian latent representation $(S_i, S_j)$1, with mean vector $(S_i, S_j)$2 and log-variance vector $(S_i, S_j)$3. The decoder is an LSTM RNN conditioned on $(S_i, S_j)$4, and its hidden state is split into two branches (Li et al., 2018). The generative branch reconstructs the input sketch sequence using a mixture of Gaussians for $(S_i, S_j)$5 and a categorical distribution for pen state. The discriminative branch produces a 128D feature vector $(S_i, S_j)$6 for each segment, which is then used to predict pairwise affinities.

Pairwise grouping is formulated through an absolute element-wise feature difference,

$(S_i, S_j)$7

followed by a small classifier yielding predicted affinity

$(S_i, S_j)$8

This defines the local grouping mechanism, but the model is explicitly designed to go beyond purely local pairwise decisions by imposing global structure on the full predicted affinity matrix $(S_i, S_j)$9 (Li et al., 2018).

The architecture therefore combines stochastic global sketch encoding, explicit per-segment embeddings, and affinity-based grouping prediction within a single recurrent generative-discriminative system. This design reflects the premise that grouping should benefit from a latent representation rich enough to encode geometry, stroke continuity, and spatial layout, rather than relying only on a narrowly discriminative embedding.

4. Generative and discriminative losses

The GroupSketch training objective is

$\mathbf{G} \in \mathbb{R}^{N \times N}$0

where $\mathbf{G} \in \mathbb{R}^{N \times N}$1 is the local grouping loss, $\mathbf{G} \in \mathbb{R}^{N \times N}$2 is the global grouping loss, $\mathbf{G} \in \mathbb{R}^{N \times N}$3 is the reconstruction loss, and $\mathbf{G} \in \mathbb{R}^{N \times N}$4 is the VAE KL divergence term (Li et al., 2018).

The reconstruction loss is the negative log-likelihood of the observed stroke sequence under the decoder’s predicted distributions: $\mathbf{G} \in \mathbb{R}^{N \times N}$5 The KL term regularizes the latent space against a standard Gaussian prior. In the paper’s interpretation, these generative losses improve generalization to unseen categories and domains by forcing the model to learn a richer latent representation than grouping alone would require (Li et al., 2018).

The local grouping loss is a binary cross-entropy over all predicted affinities $\mathbf{G} \in \mathbb{R}^{N \times N}$6 relative to the ground-truth affinity matrix $\mathbf{G} \in \mathbb{R}^{N \times N}$7. Its role is to encourage same-group segments to have similar features and different-group segments to have distinct features. However, the paper emphasizes that local pairwise consistency does not by itself guarantee transitivity or coherent grouping at the sketch level.

To address that limitation, GroupSketch introduces a novel global grouping loss based on a triplet ranking loss over rows of the predicted affinity matrix. Each row $\mathbf{G} \in \mathbb{R}^{N \times N}$8 is treated as a global grouping signature for segment $\mathbf{G} \in \mathbb{R}^{N \times N}$9, capturing its relation to all other segments. For anchor $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$0, positive $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$1 from the same group, and negative $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$2 from a different group,

$$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$3

with $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$4 defined as squared Euclidean distance over $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$5-normalized vectors (Li et al., 2018). This loss is described as novel in the context of sketch grouping and is intended to enforce global grouping consistency under local noise.

The reported typical loss weights are $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$6, $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$7, and $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$8. The paper states that removing any of $$G_{i,j} = \begin{cases} 1 & \text{if segments } S_i, S_j \text{ are in the same group} \ 0 & \text{otherwise.} \end{cases}$$9, $1$0, or $1$1 significantly degrades performance, and that replacing the separate local and global losses with a simple $1$2 loss on $1$3 versus $1$4 yields the worst performance among the tested variants (Li et al., 2018). This suggests that the central methodological contribution is not merely affinity prediction, but the structured decomposition of the supervision into local and global constraints.

5. Training, inference, and empirical role

For SPG, the training split over the 20 seen categories uses 650 sketches for training, 50 for validation, and 100 for testing per category, while each of the 5 unseen categories contributes 100 testing sketches and no training data (Li et al., 2018). Optimization uses TensorFlow on a single Titan X GPU with Adam, $1$5, $1$6, $1$7, an initial learning rate of $1$8, exponential decay, 22,000 iterations, and batch size 100. Regularization includes stroke-level distortion, stroke removal, and VAE KL regularization.

At inference time, the input sketch is encoded to latent $1$9, decoded to segment features $\mathbf{G}$0 and predicted affinities $\mathbf{G}$1, and then converted into final groups by a parameter-free agglomerative clustering method cited as Yang (2016) (Li et al., 2018). The number of groups is not fixed in advance; it is estimated from the affinity structure, and no additional learned parameters are introduced at that stage.

Evaluation follows unsupervised image segmentation metrics: Variation of Information (VOI), where lower is better; Probabilistic Rand Index (PRI), where higher is better; and Segmentation Covering (SC), where higher is better (Li et al., 2018). On seen categories, the paper states that GroupSketch significantly outperforms Edge-PG, described as a hand-crafted Gestalt-based grouper, and DeepLab, described as a deep semantic segmentation model, with improvements in VOI, PRI, and SC across all 20 categories. On unseen categories, the model is trained only on seen categories and tested directly on Angel, Bulldozer, Drill, Flower, and House, thereby using the seen/unseen split as an empirical test of universality.

The paper also states that the grouper is useful for multiple downstream tasks, including sketch synthesis and fine-grained sketch-based image retrieval (FG-SBIR) (Li et al., 2018). Because the model predicts parts rather than semantic part names, its value lies in providing a reusable structural decomposition that can support subsequent sketch analysis modules without requiring category-specific part ontologies.

6. Later reuse of the name: vector sketch animation and multi-level representations

In 2025, the name GroupSketch was reused for a different problem: vector sketch animation from text prompts in multi-object scenes (Liang et al., 21 Aug 2025). This later GroupSketch is a two-stage pipeline consisting of Motion Initialization and Motion Refinement. In the first stage, the input sketch is interactively divided into semantic groups, and users define keyframes for each group. Intermediate motions are generated by linear interpolation,

$\mathbf{G}$2

producing a coarse animation trajectory for each group (Liang et al., 21 Aug 2025). In the second stage, a Group-based Displacement Network (GDN) refines the animation by predicting group-specific displacement fields guided by a frozen text-to-video model through Score Distillation Sampling (SDS).

The later GroupSketch retains the core importance of grouping, but the role of grouping changes. In the 2018 system, grouping is the prediction target. In the 2025 system, grouping is an explicit input structure that decouples independently moving objects and supports context-aware motion refinement. The refinement network includes Context-conditioned Feature Enhancement (CCFE), Frame-aware Positional Encoding (FPE), and Motion Context Learning (MCL), and its displacement field combines a local path and a global path for each semantic group (Liang et al., 21 Aug 2025).

A related 2025 paper, SDGraph, does not define a system named GroupSketch, but it places grouping-oriented sketch reasoning in a broader representational framework by organizing sketches into sketch-level, stroke-level, and point-level information, with a Sparse Graph over strokes and a Dense Graph over points (Cheng et al., 14 Oct 2025). In that account, effective information at stroke level includes intra-stroke information and inter-stroke relations, while inter-stroke temporal information is reported as unhelpful for the studied tasks. The paper explicitly remarks that this perspective informs how strokes should be grouped: by geometry and spatial relations rather than drawing order (Cheng et al., 14 Oct 2025).

Taken together, these works show that “GroupSketch” names two different systems but one recurring research theme. In the original 2018 sense, GroupSketch is a universal perceptual grouping model for free-hand sketches. In the later 2025 sense, it is a grouping-aware animation framework. A plausible implication is that grouping has become a stable intermediate abstraction for vector sketches: it can function either as the output of sketch understanding or as the input scaffold for downstream generative control.