SketchAnimator: Vector Sketch Animation

• SketchAnimator is a vector sketch animation model that transforms a single static sketch into an animated sequence using motion cues from a reference video.
• It employs a three-stage process—Appearance Learning, Motion Learning, and Video Prior Distillation—leveraging LoRA adaptation and SDS optimization over degree-3 Bézier curves.
• The method achieves high fidelity in appearance preservation, motion alignment, and temporal consistency, offering a robust solution for reference-driven sketch animations.

SketchAnimator is a sketch animation model for motion customization of text-to-video diffusion models. Its task is to take a single static sketch and a reference video and generate a sketch-style video in which the original sketch is animated to follow the motion pattern of the reference. The method divides sketch animation into three stages—Appearance Learning, Motion Learning, and Video Prior Distillation—and combines LoRA adaptation with Score Distillation Sampling (SDS) over degree-3 Bézier curves. The stated objective is to preserve the original appearance of the sketch while transferring the dynamic movements of the reference video under one-shot motion customization (Yang et al., 10 Aug 2025).

1. Task definition and research setting

The problem addressed by SketchAnimator is narrower than generic text-to-video generation and broader than conventional sketch interpolation. It assumes a vector-stroke sketch as the source appearance and a reference video as the source of motion. The central technical difficulties are stated explicitly: preserving the exact appearance of the user’s sketch, extracting purely the motion or dynamics from the driving video without leaking unwanted appearance cues, and doing so in a one-shot setting (Yang et al., 10 Aug 2025).

Within sketch animation research, this places SketchAnimator alongside several adjacent problem formulations. “Breathing Life Into Sketches Using Text-to-Video Priors” animates a single-subject sketch from a text prompt and produces a short animation in vector representation (Gal et al., 2023). “FlipSketch” generates raster sketch animations from a single doodle and a short text instruction (Bandyopadhyay et al., 2024). “VidSketch” generates video animations directly from any number of hand-drawn sketches and simple text prompts (Jiang et al., 3 Feb 2025). Earlier animation-oriented systems such as “SketchBetween” instead learn from rendered keyframes and sketched in-betweens to synthesize sprite animations (Loftsdóttir et al., 2022). This comparison clarifies that SketchAnimator is neither a keyframe inbetweener nor a purely text-driven animator: it is a reference-video-driven vector animation system.

A common misunderstanding is to treat all sketch animation methods as interchangeable variants of diffusion-guided video generation. SketchAnimator is more specific. Its output is not an unconstrained raster video; rather, it is obtained by optimizing the control points of Bézier strokes so that a customized diffusion prior is distilled into an editable vector animation (Yang et al., 10 Aug 2025).

2. Three-stage architecture

SketchAnimator is built on top of a frozen, pretrained text-to-video diffusion model. The system introduces two successive LoRA adaptation stages and then a final SDS-based vector optimization stage (Yang et al., 10 Aug 2025).

Stage 1: Appearance Learning. The goal is to teach the frozen video diffusion model to reproduce the appearance of the input sketch as a single subject. The implementation freezes the original model weights $W_0$, adds low-rank adapters $\Delta W_A$ in the spatial attention layers only, and optimizes these adapters under the denoising objective

$$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$

The appearance prompt is represented as a semantic noun matching the sketch, exemplified in the description by $y_a =$ “A horse.” (Yang et al., 10 Aug 2025).

Stage 2: Motion Learning. A second set of LoRA adapters, $\Delta W_M$, is introduced to absorb motion dynamics from the reference video. These adapters are injected into both the spatial and temporal attention blocks, while the appearance adapters are kept fixed. Motion learning uses a spatial loss on a randomly chosen frame and a temporal loss on the full video clip, with the temporal term written as

$$L_{\mathrm{temporal}} = E_{z_0,y_m,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_M}(z_t,\tau(y_m),t) \right\|^2.$$

The combined motion objective is

$$L_{\mathrm{motion}} = L_{\mathrm{spatial}} + L_{\mathrm{temporal}}.$$

The stated purpose is to capture motion without overwriting the sketch appearance already learned in Stage 1 (Yang et al., 10 Aug 2025).

Stage 3: Video Prior Distillation. The final stage converts the customized text-to-video prior $W_0+\Delta W_A+\Delta W_M$ into an actual sketch animation by optimizing the Bézier-curve control points across frames. At this stage, the diffusion model is no longer being adapted; instead, the animation parameters are updated so that the rasterized sketch video matches the learned appearance-and-motion prior through SDS (Yang et al., 10 Aug 2025).

This staged design is structurally different from single-stage SDS methods. “Breathing Life Into Sketches Using Text-to-Video Priors” directly distills a frozen text-to-video prior into a local-plus-global vector displacement model from text alone (Gal et al., 2023). SketchAnimator inserts an intermediate customization step in which appearance and motion are separately encoded into the prior before vector optimization (Yang et al., 10 Aug 2025).

3. Vector parameterization and diffusion-based optimization

The geometric representation in SketchAnimator is explicitly vectorial. Each stroke is parameterized as a degree-3 Bézier curve

$$B(\tau)=\sum_{i=0}^3 {3 \choose i}(1-\tau)^{3-i}\tau^i P_i, \qquad P_i\in\mathbb{R}^2.$$

The input sketch contains $N$ strokes, and these strokes are duplicated across $\Delta W_A$0 frames so that all frames are initially identical. Let $\Delta W_A$1 denote all 2D control points across frames (Yang et al., 10 Aug 2025).

At each SDS iteration, the current vector animation is rasterized via a differentiable vector-graphics rasterizer $\Delta W_A$2 to produce an image $\Delta W_A$3, encoded to a latent $\Delta W_A$4, noised to $\Delta W_A$5, and evaluated by the customized prior $\Delta W_A$6. The loss follows DreamFusion-style SDS: $\Delta W_A$7 with gradient

$\Delta W_A$8

The paper also writes the effective customized weight set as

$\Delta W_A$9

This formulation makes the editable vector geometry, rather than the diffusion model, the final optimization target (Yang et al., 10 Aug 2025).

The LoRA parameterization itself is written as

$$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$0

followed by

$$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$1

In effect, SketchAnimator uses low-rank updates to specialize the motion prior and then uses SDS to transfer that specialization into Bézier control-point trajectories (Yang et al., 10 Aug 2025).

This vector-centric design contrasts with raster-frame systems such as FlipSketch, which fine-tunes a text-to-video model in latent video space and decodes final latents to raster frames (Bandyopadhyay et al., 2024), and with trajectory-centric vector systems such as the differentiable motion trajectory formulation that represents stroke control points as polynomial trajectories across time (Zhu et al., 30 Sep 2025).

4. Data sources, optimization schedule, and computational profile

The reported data sources for SketchAnimator are divided by role. Motion transfer videos come from the MGIF dataset, while sketch sources come from CLIPasso, QuickDraw, and SketchVOS (5 sketches per video) (Yang et al., 10 Aug 2025).

The training and optimization schedule is compact but staged. Appearance LoRA and Motion LoRA tuning each run for 500 iterations, using the Adam optimizer on one RTX 3090. For the SDS stage that updates curve control points, the reported learning rate is $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$2. The LoRA scales are given as $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$3 for appearance and $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$4 for motion (Yang et al., 10 Aug 2025).

The computational profile is explicitly asymmetric across stages. Appearance and motion LoRA fine-tuning take a few minutes each, whereas SDS curve optimization per clip takes on the order of 1–2 hours, depending on frame count and sampling schedule (Yang et al., 10 Aug 2025). This suggests that the method belongs to the optimization-heavy branch of sketch animation systems rather than the feed-forward branch. Comparable optimization-oriented systems include the text-guided Bézier method with runtime $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$5 h/sequence (Rai et al., 2024), the two-stage multi-object vector system with $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$6 min/video on a V100 GPU (Liang et al., 21 Aug 2025), and MoSketch with $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$7 SDS steps ($$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$8 hour on RTX 3090 Ti) (Liu et al., 25 Mar 2025).

The hardware footprint is correspondingly modest at the adaptation stage and substantial at the final distillation stage. A plausible implication is that SketchAnimator is better suited to offline authoring than to live preview, which is consistent with the paper’s explicit remark that real-time sketch preview remains open (Yang et al., 10 Aug 2025).

5. Evaluation, ablations, and observed behavior

The reported evaluation uses three metrics: Appearance Alignment, defined as average CLIP-image cosine between generated frames and the input sketch; Motion Alignment, defined as X-CLIP Score between generated video and motion prompt; and Temporal Consistency, defined as average frame-to-frame CLIP similarity (Yang et al., 10 Aug 2025).

Method	Appearance ↑	Motion ↑	Temporal ↑
FOMM	0.824	0.209	0.934
Custom-A-Video	0.689	0.314	0.879
MotionDirector	0.729	0.398	0.942
DreamVideo	0.743	0.407	0.951
Live Sketch	0.948	0.460	0.980
SketchAnimator	0.955	0.541	0.988

On this table excerpt, SketchAnimator records the highest values on all three reported axes (Yang et al., 10 Aug 2025). The qualitative summary states that competing methods either deform the sketch or produce nearly static outputs, whereas SketchAnimator preserves shape while injecting richer motion (Yang et al., 10 Aug 2025).

The architectural ablation is equally informative. The full model is compared with direct control-point optimization and with variants that remove the global or local branch:

Variant	Sketch-video	Text-video
Full model	0.965	0.142
No network (directly optimize control points)	0.926	0.142
No global branch	0.936	0.140
No local branch	0.970	0.140

The no local branch variant attains a slightly higher sketch-video score but is reported to produce unreal wobble, indicating that the scalar metric alone does not capture motion plausibility (Yang et al., 10 Aug 2025). The user study involved 31 participants over 30 pairs in a forced-choice comparison, with the full model preferred $$L_{\mathrm{appearance}} = E_{z_0,y_a,t,\epsilon} \left\| \epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t) \right\|^2.$$9 of the time on fidelity and $y_a =$0 of the time on alignment over every ablation (Yang et al., 10 Aug 2025). The examples are described as covering animals, humans, and objects, and the same sketch can be animated by different prompts such as a boxer “punching” versus “dodging” (Yang et al., 10 Aug 2025).

6. Relation to neighboring methods, scope, and limitations

SketchAnimator occupies a specific location in the current sketch-animation landscape. It is a single-sketch, single-reference-video, vector-output system. Other directions in the literature emphasize different control modalities and output spaces. “SketchBetween” addresses rendered keyframes and sketched in-betweens for sprite animation (Loftsdóttir et al., 2022). “Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening” addresses automatic interpolation between two sketch keyframes with multi-level guidance and a multi-stream U-Transformer (Shen et al., 2023). “MoSketch” and “GroupSketch” target multi-object sketch animation with scene decomposition, motion planning, group assignment, and refinement networks (Liu et al., 25 Mar 2025, Liang et al., 21 Aug 2025). “Sketch2Anim” translates storyboard sketches into 3D motions (Zhong et al., 27 Apr 2025), while “Sketch2Colab” extends sketch-conditioned animation into multi-human and object-aware 3D motion with controllable constraints (Daiya et al., 2 Mar 2026). “Notational Animating” shifts the problem from automatic motion transfer to an interactive authoring paradigm based on sketched motion notation and a closed feedback loop (Shi et al., 6 Mar 2026).

These comparisons clarify two points. First, SketchAnimator is not a general solution to all sketch animation settings. It does not address the multi-object divide-and-conquer strategies of MoSketch or the group-specific displacement refinement of GroupSketch (Liu et al., 25 Mar 2025, Liang et al., 21 Aug 2025). Second, it does not replace interactive authoring systems in which ambiguity is surfaced and corrected explicitly through UI widgets or timeline edits (Shi et al., 6 Mar 2026).

The limitations stated for SketchAnimator are specific and consequential. Very complex or highly non-rigid motions may not transfer cleanly from a single reference clip. Extremely exotic sketch styles may require more LoRA iterations or multiple appearance examples. The pipeline remains relatively slow at the final SDS optimization, and real-time sketch preview remains open. The authors identify multi-object scenes, interactive user control of key frames, and 3D consistency as promising directions (Yang et al., 10 Aug 2025).

In the provided corpus, the name “SketchAnimator” also appears in implementation-oriented summaries as a generic label for sketch-driven animation tools built from other models. This suggests a broader descriptive use of the term. In the stricter bibliographic sense, however, SketchAnimator denotes the 2025 system that combines appearance LoRA, motion LoRA, and SDS optimization of Bézier curves for one-shot motion customization of text-to-video diffusion models (Yang et al., 10 Aug 2025).