---
title: 'SketchAnimator: Vector Sketch Animation'
url: https://www.emergentmind.com/topics/sketchanimator
type: topic
---

# SketchAnimator: Vector Sketch Animation

SketchAnimator is a sketch animation model for **motion customization of text-to-video diffusion models**. Its task is to take **a single static sketch** and **a reference video** and generate **a sketch-style video** in which the original sketch is animated to follow the motion pattern of the reference. The method divides sketch animation into **three stages**—**Appearance Learning**, **Motion Learning**, and **Video Prior Distillation**—and combines **LoRA** adaptation with **Score Distillation Sampling (SDS)** over **degree-3 Bézier curves**. The stated objective is to preserve the original appearance of the sketch while transferring the dynamic movements of the reference video under **one-shot motion customization** [2508.07149].

## 1. Task definition and research setting

The problem addressed by SketchAnimator is narrower than generic text-to-video generation and broader than conventional sketch interpolation. It assumes a **vector-stroke sketch** as the source appearance and a **reference video** as the source of motion. The central technical difficulties are stated explicitly: preserving the exact appearance of the user’s sketch, extracting purely the motion or dynamics from the driving video without leaking unwanted appearance cues, and doing so in a one-shot setting [2508.07149].

Within sketch animation research, this places SketchAnimator alongside several adjacent problem formulations. “Breathing Life Into Sketches Using Text-to-Video Priors” animates a **single-subject sketch** from a **text prompt** and produces a short animation in **vector representation** [2311.13608]. “FlipSketch” generates **raster sketch animations** from **a single doodle and a short text instruction** [2411.10818]. “VidSketch” generates video animations directly from **any number of hand-drawn sketches and simple text prompts** [2502.01101]. Earlier animation-oriented systems such as “SketchBetween” instead learn from **rendered keyframes and sketched in-betweens** to synthesize sprite animations [2209.00185]. This comparison clarifies that SketchAnimator is neither a keyframe inbetweener nor a purely text-driven animator: it is a **reference-video-driven vector animation system**.

A common misunderstanding is to treat all sketch animation methods as interchangeable variants of diffusion-guided video generation. SketchAnimator is more specific. Its output is not an unconstrained raster video; rather, it is obtained by optimizing the control points of Bézier strokes so that a customized diffusion prior is distilled into an editable vector animation [2508.07149].

## 2. Three-stage architecture

SketchAnimator is built on top of a **frozen, pretrained text-to-video diffusion model**. The system introduces **two successive LoRA adaptation stages** and then a final SDS-based vector optimization stage [2508.07149].

**Stage 1: Appearance Learning.** The goal is to teach the frozen video diffusion model to reproduce the appearance of the input sketch as a single subject. The implementation freezes the original model weights \(W_0\), adds low-rank adapters \(\Delta W_A\) in the **spatial attention** layers only, and optimizes these adapters under the denoising objective
\[
L_{\mathrm{appearance}}
=
E_{z_0,y_a,t,\epsilon}
\left\|
\epsilon-\epsilon_{W_0+\Delta W_A}(z_t,\tau(y_a),t)
\right\|^2.
\]
The appearance prompt is represented as a semantic noun matching the sketch, exemplified in the description by \(y_a =\) “A horse.” [2508.07149].

**Stage 2: Motion Learning.** A second set of LoRA adapters, \(\Delta W_M\), is introduced to absorb motion dynamics from the reference video. These adapters are injected into **both the spatial and temporal attention blocks**, while the appearance adapters are kept fixed. Motion learning uses a **spatial loss** on a randomly chosen frame and a **temporal loss** on the full video clip, with the temporal term written as
\[
L_{\mathrm{temporal}}
=
E_{z_0,y_m,t,\epsilon}
\left\|
\epsilon-\epsilon_{W_0+\Delta W_M}(z_t,\tau(y_m),t)
\right\|^2.
\]
The combined motion objective is
\[
L_{\mathrm{motion}} = L_{\mathrm{spatial}} + L_{\mathrm{temporal}}.
\]
The stated purpose is to capture motion without overwriting the sketch appearance already learned in Stage 1 [2508.07149].

**Stage 3: Video Prior Distillation.** The final stage converts the customized text-to-video prior \(W_0+\Delta W_A+\Delta W_M\) into an actual sketch animation by optimizing the Bézier-curve control points across frames. At this stage, the diffusion model is no longer being adapted; instead, the animation parameters are updated so that the rasterized sketch video matches the learned appearance-and-motion prior through SDS [2508.07149].

This staged design is structurally different from single-stage SDS methods. “Breathing Life Into Sketches Using Text-to-Video Priors” directly distills a frozen text-to-video prior into a local-plus-global vector displacement model from text alone [2311.13608]. SketchAnimator inserts an intermediate customization step in which **appearance** and **motion** are separately encoded into the prior before vector optimization [2508.07149].

## 3. Vector parameterization and diffusion-based optimization

The geometric representation in SketchAnimator is explicitly vectorial. Each stroke is parameterized as a **degree-3 Bézier curve**
\[
B(\tau)=\sum_{i=0}^3 {3 \choose i}(1-\tau)^{3-i}\tau^i P_i, \qquad P_i\in\mathbb{R}^2.
\]
The input sketch contains \(N\) strokes, and these strokes are duplicated across \(F\) frames so that all frames are initially identical. Let \(\phi\) denote all 2D control points across frames [2508.07149].

At each SDS iteration, the current vector animation is rasterized via a **differentiable vector-graphics rasterizer** \(R_\phi\) to produce an image \(x\), encoded to a latent \(z_0 = E(x)\), noised to \(z_t\), and evaluated by the customized prior \(\epsilon_{W_0+\Delta W_A+\Delta W_M}(z_t,\tau(y_{\mathrm{motion}}),t)\). The loss follows DreamFusion-style SDS:
\[
L_{\mathrm{SDS}}
=
E_{t,\epsilon}\left[w(t)\left\|\epsilon_\theta(z_t,\tau(y),t)-\epsilon\right\|^2\right],
\]
with gradient
\[
\nabla_\phi L_{\mathrm{SDS}}
=
E_{t,\epsilon}
\left[
w(t)\cdot
(\epsilon_\theta(z_t,\tau(y),t)-\epsilon)\cdot \frac{\partial x}{\partial \phi}
\right].
\]
The paper also writes the effective customized weight set as
\[
W' = W_0 + \lambda_1 \Delta W_A + \lambda_2 \Delta W_M.
\]
This formulation makes the editable vector geometry, rather than the diffusion model, the final optimization target [2508.07149].

The LoRA parameterization itself is written as
\[
\Delta W = B\cdot A, \qquad
B\in\mathbb{R}^{d\times r}, \;
A\in\mathbb{R}^{r\times k}, \;
r\ll \min(d,k),
\]
followed by
\[
W = W_0 + \alpha \cdot \Delta W.
\]
In effect, SketchAnimator uses low-rank updates to specialize the motion prior and then uses SDS to transfer that specialization into Bézier control-point trajectories [2508.07149].

This vector-centric design contrasts with raster-frame systems such as FlipSketch, which fine-tunes a text-to-video model in latent video space and decodes final latents to raster frames [2411.10818], and with trajectory-centric vector systems such as the differentiable motion trajectory formulation that represents stroke control points as polynomial trajectories across time [2509.25857].

## 4. Data sources, optimization schedule, and computational profile

The reported data sources for SketchAnimator are divided by role. **Motion transfer videos** come from the **MGIF dataset**, while sketch sources come from **CLIPasso, QuickDraw, and SketchVOS (5 sketches per video)** [2508.07149].

The training and optimization schedule is compact but staged. **Appearance LoRA** and **Motion LoRA** tuning each run for **500 iterations**, using the **Adam optimizer** on **one RTX 3090**. For the SDS stage that updates curve control points, the reported learning rate is **\(2\times10^{-3}\)**. The LoRA scales are given as **\(\lambda_1 = 0.5\)** for appearance and **\(\lambda_2 = 1.0\)** for motion [2508.07149].

The computational profile is explicitly asymmetric across stages. Appearance and motion LoRA fine-tuning take **a few minutes each**, whereas SDS curve optimization per clip takes **on the order of 1–2 hours**, depending on frame count and sampling schedule [2508.07149]. This suggests that the method belongs to the optimization-heavy branch of sketch animation systems rather than the feed-forward branch. Comparable optimization-oriented systems include the text-guided Bézier method with **runtime \(\sim 2\) h/sequence** [2411.19381], the two-stage multi-object vector system with **\(\sim 40\) min/video** on a V100 GPU [2508.15535], and MoSketch with **\(T=500\) SDS steps (\(\sim 1\) hour on RTX 3090 Ti)** [2503.19351].

The hardware footprint is correspondingly modest at the adaptation stage and substantial at the final distillation stage. A plausible implication is that SketchAnimator is better suited to offline authoring than to live preview, which is consistent with the paper’s explicit remark that **real-time sketch preview remains open** [2508.07149].

## 5. Evaluation, ablations, and observed behavior

The reported evaluation uses three metrics: **Appearance Alignment**, defined as average **CLIP-image cosine** between generated frames and the input sketch; **Motion Alignment**, defined as **X-CLIP Score** between generated video and motion prompt; and **Temporal Consistency**, defined as average **frame-to-frame CLIP similarity** [2508.07149].

| Method | Appearance ↑ | Motion ↑ | Temporal ↑ |
|---|---:|---:|---:|
| FOMM | 0.824 | 0.209 | 0.934 |
| Custom-A-Video | 0.689 | 0.314 | 0.879 |
| MotionDirector | 0.729 | 0.398 | 0.942 |
| DreamVideo | 0.743 | 0.407 | 0.951 |
| Live Sketch | 0.948 | 0.460 | 0.980 |
| SketchAnimator | 0.955 | 0.541 | 0.988 |

On this table excerpt, SketchAnimator records the highest values on all three reported axes [2508.07149]. The qualitative summary states that competing methods either **deform the sketch** or produce **nearly static outputs**, whereas SketchAnimator preserves shape while injecting richer motion [2508.07149].

The architectural ablation is equally informative. The full model is compared with direct control-point optimization and with variants that remove the global or local branch:

| Variant | Sketch-video | Text-video |
|---|---:|---:|
| Full model | 0.965 | 0.142 |
| No network (directly optimize control points) | 0.926 | 0.142 |
| No global branch | 0.936 | 0.140 |
| No local branch | 0.970 | 0.140 |

The **no local branch** variant attains a slightly higher sketch-video score but is reported to produce **unreal wobble**, indicating that the scalar metric alone does not capture motion plausibility [2508.07149]. The user study involved **31 participants over 30 pairs** in a forced-choice comparison, with the full model preferred **\(\ge 75\%\)** of the time on fidelity and **\(\ge 50\%\)** of the time on alignment over every ablation [2508.07149]. The examples are described as covering **animals, humans, and objects**, and the same sketch can be animated by different prompts such as a boxer “**punching**” versus “**dodging**” [2508.07149].

## 6. Relation to neighboring methods, scope, and limitations

SketchAnimator occupies a specific location in the current sketch-animation landscape. It is a **single-sketch**, **single-reference-video**, **vector-output** system. Other directions in the literature emphasize different control modalities and output spaces. “SketchBetween” addresses **rendered keyframes and sketched in-betweens** for sprite animation [2209.00185]. “Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening” addresses automatic interpolation between two sketch keyframes with **multi-level guidance** and a **multi-stream U-Transformer** [2308.13273]. “MoSketch” and “GroupSketch” target **multi-object sketch animation** with scene decomposition, motion planning, group assignment, and refinement networks [2503.19351; 2508.15535]. “Sketch2Anim” translates **storyboard sketches into 3D motions** [2504.19189], while “Sketch2Colab” extends sketch-conditioned animation into **multi-human** and **object-aware 3D motion** with controllable constraints [2603.02190]. “Notational Animating” shifts the problem from automatic motion transfer to an **interactive authoring paradigm** based on sketched motion notation and a closed feedback loop [2603.06880].

These comparisons clarify two points. First, SketchAnimator is not a general solution to all sketch animation settings. It does not address the multi-object divide-and-conquer strategies of MoSketch or the group-specific displacement refinement of GroupSketch [2503.19351; 2508.15535]. Second, it does not replace interactive authoring systems in which ambiguity is surfaced and corrected explicitly through UI widgets or timeline edits [2603.06880].

The limitations stated for SketchAnimator are specific and consequential. **Very complex or highly non-rigid motions** may not transfer cleanly from a single reference clip. **Extremely exotic sketch styles** may require more LoRA iterations or multiple appearance examples. The pipeline remains **relatively slow at the final SDS optimization**, and **real-time sketch preview remains open**. The authors identify **multi-object scenes**, **interactive user control of key frames**, and **3D consistency** as promising directions [2508.07149].

In the provided corpus, the name “SketchAnimator” also appears in implementation-oriented summaries as a generic label for sketch-driven animation tools built from other models. This suggests a broader descriptive use of the term. In the stricter bibliographic sense, however, SketchAnimator denotes the 2025 system that combines **appearance LoRA**, **motion LoRA**, and **SDS optimization of Bézier curves** for **one-shot motion customization of text-to-video diffusion models** [2508.07149].

Source: https://www.emergentmind.com/topics/sketchanimator