ToonComposer: Unified Cartoon Animation

• ToonComposer is an integrated model that unifies inbetweening and colorization to streamline cartoon video production.
• It employs innovations like sparse sketch injection and spatial low-rank adaptation to reduce manual errors and ensure temporal coherence.
• Empirical benchmarks show superior visual fidelity and smooth motion, providing robust artist control for domain-specific animation.

ToonComposer is a generative model for cartoon and anime video production that unifies the inbetweening (animation frame interpolation) and colorization (appearance rendering) stages into a single, integrated post-keyframing process. Developed to address the manual bottlenecks and error propagation endemic to traditional cel animation pipelines, ToonComposer achieves high-quality, temporally coherent animated sequences given only a sparse set of artist-drawn sketches and colored reference frames. Its design leverages innovations in sparse sketch injection and spatial low-rank adaptation, facilitating both artist-controllable output and domain transfer to the unique stylistic requirements of cartoon content (Li et al., 14 Aug 2025).

1. Integrative Production Pipeline

Traditional cartoon production separates keyframing, inbetweening, and colorization, often requiring hundreds of hand-drawn frames with dense lineart and error-prone, sequential processing. Preceding AI methods typically address these tasks independently, resulting in problems such as motion artifacts or color/style inconsistency when intermediate outputs are propagated across stages.

ToonComposer unifies inbetweening and colorization into a single post-keyframing generative stage. Given as few as a single keyframe sketch and a colored reference frame, ToonComposer can generate full-length sequences that preserve the intended motion and stylistic attributes of the original artwork. This approach eliminates error accumulation and facilitates a substantially more flexible artist workflow: multiple sketches and color cues can be injected at arbitrary frames, supporting both minimal and highly controlled authoring scenarios.

2. Sparse Sketch Injection Mechanism

The model’s sparse sketch injection mechanism enables efficient and precise conditioning on limited user input:

• Keyframe sketches (and, if available, their corresponding color references) are encoded into latent embeddings via a learned projection head.
• These embeddings are mapped into the latent space of a DiT-based video diffusion model at the temporally appropriate indices, using injected positional encodings equivalent to the temporal location within the sequence.
• A position-aware residual module linearly transforms the injected sketch tokens with a trainable weight matrix $W_{res}$ and adds them to the associated video latent tokens, modulated by a scaling parameter $\alpha$. During training, $\alpha=1$; at inference, it is user-adjustable, granting control over the influence of each sketch constraint:

$${z_k^{(t)}}[\text{indices } i_n] := {z_k^{(t)}}[\text{indices } i_n] + \alpha \left( s'_{i_n} W_{res} \right)$$

This allows arbitrary numbers of sketches and color references, enabling region-wise control. If users annotate only part of the scene, ToonComposer can plausibly inpaint the remainder, guided by both context and global style.

3. Cartoon Domain Adaptation: Spatial Low-Rank Adapter

ToonComposer employs a Spatial Low-Rank Adapter (SLRA), a mechanism for domain adaptation that is compatible with the full spatiotemporal attention of modern video DiT architectures:

• Hidden feature maps of dimension $D$ are down-projected with a trainable matrix $W^{(down)} \in \mathbb{R}^{D \times D'}$ ($D' \ll D$), yielding $h^{(low)}$.
• Features are rearranged to their spatial-temporal arrangement and a self-attention operation is performed across the spatial dimension, preserving temporal priors ($h' = \text{SelfAttention}(h) + \text{SLRA}(h)$).
• The resulting features are upsampled and added as a residual to the original self-attention output, permitting efficient learning of cartoon-specific appearance (line art, brush style, palette) with minimal adaptation of dynamic (temporal) structure.

This preservation of temporal priors is essential for continuity and smooth motion, while spatial adaptation affords genre-specific rendering consistency.

4. Performance and Benchmarking

To facilitate rigorous evaluation, the authors introduce PKBench, a new in-the-wild benchmark dataset featuring human-drawn sketches and colored reference frames sourced to mimic real-world cartoon production scenarios.

Empirical results establish ToonComposer’s state-of-the-art performance across multiple criteria:

• On synthetic benchmarks, ToonComposer achieves LPIPS of 0.1785 (compared to baselines around 0.37–0.39), indicating closer perceptual similarity to ground truth. It also attains lower DISTS and higher CLIP similarity.
• PKBench evaluations show superiority in subject consistency, motion smoothness, background consistency, and overall video aesthetics. Both reference-based metrics and user studies (win rates for preference) robustly favor ToonComposer over prior models (such as AniDoc, LVCD, and ToonCrafter).
• Qualitative results on professional cartoon/film clips indicate robustness to challenging motions (e.g., fast facial deformations, complex choreography) and retention of coherent style.

5. Artist Control and Region-Wise Guidance

The sparse sketch injection paradigm offers fine-grained artist control:

• Artists may input minimal keyframes for maximal automation or annotate critical frames/regions for structural or stylistic fidelity.
• At inference, the $\alpha$ parameter for sketch influence can be varied to smoothly interpolate between strict adherence to the artist’s guidance and flexible AI-driven interpolation.
• Support for region-wise masking enables targeted editing (e.g., specifying only facial expressions or character outlines), making it possible to iteratively refine motion or details without re-sketching entire frames.
• Region-level blank input is interpreted as a context-guided inpainting challenge, filled in via the model’s learned priors.

6. Broader Applications and Implications

ToonComposer streamlines commercial and independent cartoon video production by reducing manual frame authoring and colorization, while allowing creators to maintain creative sovereignty via injection of precise cues. Its domain-adaptive architecture (SLRA) is applicable to other stylized video tasks, and the model supports extension to related illustration or even 3D-rendered domains (with appropriate fine-tuning).

Demonstrated use cases include professional cartoon movie scene recreation (e.g., “Big Fish” and “Begonia”), efficient prototyping, and educational content production. The ability to generalize from minimal cues suggests significant labor reductions for studios and individual artists alike.

7. Limitations and Future Research

While ToonComposer resolves major bottlenecks in cartoon video generation, certain challenges and directions remain:

• Optimizing computational efficiency—diffusion models remain resource intensive for long sequences or high resolutions.
• Enhancing control granularity and interpretability—for instance, hierarchical or attribute-specific motion guidance.
• Extending SLRA or analogous modules to broader genres, non-cartoon artistic domains, or live-action stylization.
• Generalization to complex interactive, narrative, or user-steered animation pipelines.
• Investigating the limits of minimal input—quantifying fidelity as a function of sketch sparsity and reference color adequacy—remains a valued area for further paper.

In summary, ToonComposer delivers a unified, artist-controllable generative architecture for cartoon animation, surpassing previous solutions in visual quality, motion consistency, and operational efficiency. Its integration of sparse sketch injection and spatial low-rank adaptation mechanisms establishes a new technical paradigm for AI-assisted cartoon and animation production (Li et al., 14 Aug 2025).