FairyGen: Automated Cartoon Video Generation

• FairyGen is a system that automatically converts single child-drawn sketches into multi-shot, cinematic cartoon videos with coherent narratives.
• It integrates multimodal LLMs, style propagation adapters, and 3D proxy-based motion to achieve realistic, aesthetically consistent animations.
• A two-stage motion customization and image-to-video diffusion framework preserves character identity and facilitates dynamic camera movements.

FairyGen is an automatic system designed for generating narratively coherent, stylized cartoon videos from a single child-drawn character image. Distinct from conventional visual storytelling pipelines, FairyGen explicitly disentangles the processes of character modeling and stylized background synthesis, and employs cinematic shot planning and physically plausible motion control, enabling expressive, multi-shot animation faithful to the unique aesthetic of a child’s artwork. By integrating multimodal LLMs (MLLM), adapter-based style propagation, 3D proxy-based motion, and an MMDiT-based image-to-video diffusion framework, FairyGen supports automatic, end-to-end conversion from individual sketches to structured, cinematic cartoon video (Zheng et al., 26 Jun 2025).

1. System Pipeline and Modular Architecture

FairyGen operates through a modular, sequential pipeline:

1. Input Acquisition: The system receives a single scanned or digital image $I$ representing a child-drawn character.
2. Storyboard Generation (MLLM): A multimodal LLM is prompted in two stages (“Action Planning” and “Multi-Shot Planning”) to output a structured storyboard comprising shot-level descriptions, including environment ($\text{bg}$), character action, camera type, and shot bounding box coordinates. The format is a JSON-like dictionary encoding per-shot scene and motion (Zheng et al., 26 Jun 2025).
3. Style Propagation Adapter: Using an SDXL latent-diffusion UNet backbone, the system adapts a child’s foreground style to the background. DoRA-based low-rank adapters are used for this propagation, guided by foreground (character) regions during training and background regions at inference.
4. Cinematic Shot Design: The shot module interprets bounding boxes and camera parameters from the storyboard, implements diverse cropping, and supports multi-view synthesis through rendering the 3D proxy of the character from different angles (yaw/pitch). Controls on camera focal length, crop scale, and jitter are exposed; LPIPS-consistency regularization enforces visual stability across backgrounds.
5. 3D Proxy Reconstruction and Motion Extraction: The pipeline reconstructs the 3D shape of the character using SDF-based silhouettes, rigs an articulated skeleton, and transfers motion clips mapped to story actions; smoothness and joint-limit penalties ensure plausible motion.
6. Image-to-Video Diffusion (MMDiT-based): The shot-wise outputs and pose/motion embeddings condition a multi-modal, mixed-dimension transformer diffusion model, responsible for spatio-temporal synthesis of animated video clips.
7. Two-Stage Motion Customization Adapter: LoRA adapters disentangle static identity (by training on temporally shuffled frames) and dynamic motion (by training with timestep-shift and frozen identity weights), enhancing robustness of personalized animation.

This stepwise pipeline is summarized in the following table:

Module	Purpose	Key Features
Storyboard Gen. (MLLM)	Narrative structuring/shot description	Action+shot-level planning, JSON-format output
Style Propagation Adapter	Aesthetic consistency	SDXL+DoRA, style feature matching loss
Cinematic Shot Design	Visual/cinematic diversity	Cropping, multi-view, LPIPS background consistency
3D Proxy & Motion	Physical character animation	SDF silhouette, skeleton rig, smooth motion
Diffusion Model	Video synthesis	MMDiT UNet, pose & text cross-attention
Motion Customization	Personalized motion/identity control	2-stage LoRA: identity then motion phase

2. Storyboard Generation via Multimodal LLMs

FairyGen leverages a large multimodal LLM (such as GPT-4) in a two-step prompting structure for flexible and semantically controlled narrative creation:

• Action Planning Prompt: The LLM receives a character description and generates a set of five semantically distinct story actions (e.g., “wave”, “jump”, “spin”).
• Multi-Shot Planning Prompt: For each action, the model generates shot entries specifying background ("bg"), action, camera type, and object bounding box coordinates. The standardized output enables downstream compositional and cinematic control.
• Structural Consistency: In-context demonstrations and chain-of-thought (CoT) prompting techniques regularize output structure, avoiding prompt drift. The final output is a shot-wise dictionary fed to subsequent modules for shot assembly (Zheng et al., 26 Jun 2025).

This aligns with the hierarchical script parsing and shot-level state tracking shown to increase narrative fidelity in multi-agent storyboard systems such as AnimeAgent (Yan et al., 24 Feb 2026).

3. Style and Cinematic Modules

Style Propagation Adapter

The style adapter utilizes DoRA, a weight-decomposed low-rank matrix method, inserted at every transformer block in the SDXL UNet (Zheng et al., 26 Jun 2025). During training, style is learned exclusively from tokens where foreground mask $m(x)=1$ (character regions), whereas at inference, only background tokens $(1-m)$ are adapted, applying style to the environment. Style loss is enforced using a pretrained VGG-19 feature extractor $F$:

$$L_\text{style} = \sum_l \|\mu(F_l(I_\text{char})) - \mu(F_l(I_\text{bg}))\|_2^2 + \|\sigma(F_l(I_\text{char})) - \sigma(F_l(I_\text{bg}))\|_2^2$$

This is weighted with the overall diffusion loss to yield the training objective:

$$L = L_\text{diffusion} + \lambda_\text{style} L_\text{style}$$

The module reliably transmits user-provided style to generated backgrounds, improving upon prior methods where style was poorly decoupled from motion or subject identity (He et al., 2023).

Cinematic Shot Design

• Frame Cropping and Shot Composition: The module supports aspect ratios (1:1, 4:3, 16:9), ensures bounding boxes occupy $10\%-80\%$ of frame area, and applies padding for context.
• Multi-View Synthesis: By rendering the 3D proxy at discrete yaw/pitch angles, the system generates background-consistent, multi-view sequences.
• Diversity and Regularization: Randomized camera and crop parameters, along with LPIPS-based background similarity loss, enforce both cinematic variability and cross-shot consistency.

These innovations differentiate FairyGen from autoregressive frame-wise systems, allowing frame compositionality and supporting dynamic scene transition (Liu et al., 2023).

4. Physically Plausible 3D Proxy and Motion

FairyGen reconstructs a 3D proxy mesh from the input drawing using an implicit SDF $S_\theta(p)$ optimized to match the sketch silhouette, penalized by:

$$L_\text{sil} = \sum_{p} \text{Dist}(\text{Proj}(S_\theta), \text{SketchMask}(p))$$

Skeleton rigging defines a set of joints $\text{bg}$0 embedded within the mesh. Motion clips $\text{bg}$1 corresponding to story-shot verbs are obtained from a small library and mapped onto the skeleton, with joint-limit constraints and a temporal smoothness loss:

$\text{bg}$2

Per-frame pose vectors plus translation/scale parametrize motion and are exposed as soft conditions to the video diffusion model.

This component enables physically-plausible, smooth subject animation, overcoming copy-paste artifacts typical of purely static methods (Yan et al., 24 Feb 2026).

5. Image-to-Video Synthesis and Motion Customization

Video Diffusion with MMDiT

The backbone is a multi-modal, mixed-dimension transformer (MMDiT) UNet with 3D time-axis attention and spatial convolutions. Conditioning is supplied via both text ($\text{bg}$3) and pose tokens ($\text{bg}$4). Generation employs standard Denoising Diffusion Probabilistic Models (DDPM) objectives in the latent space:

• Forward Process:

$\text{bg}$5

• Reverse Process:

$\text{bg}$6

Two-Stage Motion Customization Adapter

• Stage 1 (Identity Only): LoRA adapters $\text{bg}$7 are trained on unordered (shuffled) frames with dropout on $\text{bg}$8, capturing static appearance and disentangling it from motion.
• Stage 2 (Motion Residual): Frozen identity weights are supplemented with motion-specific adapters $\text{bg}$9, trained on ordered frames via timestep-shift sampling biased towards late time steps, which encourages the model to focus on global temporal dynamics.

This dual-adapter approach achieves robust subject identity preservation across varying pose and motion, substantially improving upon prior single-stage, static adaptation techniques (He et al., 2023).

6. Evaluation, Limitations, and Comparative Context

Quantitative Metrics

• Style Alignment (CLIP-based) Lower-is-better: Ours 0.6580, B-LoRA 0.5060, InstantStyle 0.5468, DreamBooth 0.6371.
• Motion Smoothness (VBench): Ours 0.987, Animate-X 0.974, Wan2.1 0.977.
• Subject Consistency: Ours 0.955, Animate-X 0.908, Wan2.1 0.842.
• Human evaluations: FairyGen achieves higher scores in motion realism and visual quality relative to baseline and prior state-of-the-art systems (Zheng et al., 26 Jun 2025).

Qualitative Results

Produced animations display multi-shot, cinematographically diverse sequences with style-consistent backgrounds and child-drawn character fidelity. Narrative-driven camera work (close-ups, perspective shifts), seamless complex motions (e.g., running, spinning), and expressive story progression are consistently achieved.

Limitations

• Backgrounds may remain static during character motion, causing still-background artifacts.
• Highly abstract or sparse character sketches challenge the 3D proxy reconstruction pipeline.
• Video diffusion generative prior may produce artifacts under extreme camera views.
• Open challenges include dynamic background synthesis, multi-character animation, accurate silhouette-to-3D reconstruction, and end-to-end joint tuning of narrative, motion, and style (Zheng et al., 26 Jun 2025).

Contextual Relation to Broader Visual Storytelling

FairyGen distinguishes itself from autoregressive image-sequence models like StoryGen (Liu et al., 2023) by employing a shot-structured, temporally-conditioned 3D video diffusion approach. In comparison to retrieval-augmented and control-conditioned frameworks (Animate-A-Story (He et al., 2023)), FairyGen’s pipeline emphasizes single-image subject style preservation, cinematic compositionality, and physically grounded motion synthesis. Insights from Disney-inspired, multi-agent workflows (AnimeAgent (Yan et al., 24 Feb 2026))—notably hierarchical shot parsing, iterative review, and hybrid pose-to-pose and straight-ahead animation—are reflected in FairyGen’s integration of modular story, style, and motion modules.

7. Implementation and Reproducibility

Experiments are conducted on the AnimatedDrawings dataset ($m(x)=1$0178K child-drawn inputs), with adapter fine-tuning using 24 sampled styles and a small set of custom motion videos. Training is performed on a single NVIDIA L20 GPU (16 GB):

• Style adapter: 120 min fine-tuning.
• Motion adapters: 180 min (cumulative, both stages).
• Hyperparameters include LoRA rank $m(x)=1$1, learning rate $m(x)=1$2, batch size 2, dropout $m(x)=1$3, style loss weight $m(x)=1$4, epochs: Stage 1 (200), Stage 2 (300).

The code base is publicly available at https://github.com/GVCLab/FairyGen, supporting full pipeline reconstruction from sketch input through to cinematic, personalized cartoon video output (Zheng et al., 26 Jun 2025).