SCAIL-2: End-to-End Character Animation

• SCAIL-2 is a unified, end-to-end controllable animation framework that bypasses intermediate pose and mask representations to capture detailed visual cues.
• It introduces the MotionPair-60K dataset, a large-scale synthetic resource designed for robust motion transfer, character replacement, and multi-character animation tasks.
• The framework employs innovative in-context conditioning and reverse-driving techniques to enhance cross-identity transfer and overall animation fidelity.

SCAIL-2 is a unified, end-to-end controllable character animation framework and large-scale dataset that advances motion transfer research by synthesizing raw video-based animation without reliance on intermediate pose or masked environmental representations. The framework, detailed in "SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning" (Yan et al., 9 Jun 2026), introduces new methodologies and the MotionPair-60K dataset, enabling the direct conditioning and transfer of motion, identity, and environmental context across a diverse array of animation and character replacement tasks.

1. End-to-End Character Animation Paradigm

SCAIL-2 addresses the historical dependence on pose skeletons and environment masks, which introduce information loss regarding occlusions and detailed appearance. Prior character animation systems typically represented motion via intermediate pose skeletons and handled environments using masked backgrounds. These intermediates hindered the model’s ability to recover detailed visual cues necessary for high-fidelity, robust motion transfer. SCAIL-2 bypasses these intermediates by directly concatenating the driving video to reference character imagery, allowing a video diffusion model to extract all required information directly from raw video frames. This facilitates the unification of multiple animation tasks—including single- and multi-character animation as well as cross-identity character replacement—within a single architecture (Yan et al., 9 Jun 2026).

2. MotionPair-60K Dataset Construction and Design

MotionPair-60K is a synthetic dataset specifically curated for end-to-end, cross-identity character animation. It contains heterogeneous tasks, supporting "Animation Mode" (character image animation) and "Replacement Mode" (character replacement), with pairs generated using a reverse-driving synthetic pipeline. The dataset accommodates the unification of three decoupled objectives: Motion Binding, Environment Weaving, and Universal Transfer. The scarcity of real-world end-to-end motion pairs—where multiple characters perform the same motion in an identical setting—necessitated the development of a large-scale synthetic loop that produces paired driving and target videos as well as reference images at scale. The agentic editing loop of the pipeline ensures quality and diversity through candidate selection, prompt engineering, multi-modal reference synthesis, and evaluation with a vision-LLM to filter for plausibility and quality (targeting ≥70% quality retention) (Yan et al., 9 Jun 2026).

3. Synthetic Pipeline Workflow

The SCAIL-2 pipeline proceeds as follows:

• Driving Sequence Sampling: Samples real videos $y$ containing one or more characters.
• Reference Generation via Agentic Editing Loop: Uses a candidate selector to identify a character image $I$, constructs descriptive prompts, and synthesizes or refines $I$ through a multi-reference image model (e.g., Gemini API or "Nano Banana"), followed by quality assurance and optional human-in-the-loop editing.
• Animation/Replacement Generation: A pose-driven animation model $\mathcal G$ synthesizes the prospective output $\tilde y = \mathcal G(y, I)$:
  • For Animation Mode: The SCAIL model applies robust transfer even across large body-shape discrepancies and occlusions.
  • For Replacement Mode: Utilizes a MoCha renderer-based model for character replacement.
• Reverse-Driving Construction: Swaps the synthetic result and ground truth to form training triplets $(\tilde y, I, y)$, teaching the model to reconstruct $y$ from $\tilde y$ and reference image $I$.
• Key Equations:

$$q(\mathbf z_t \mid \mathbf z_{t-1}) = \mathcal N\left(\mathbf z_t;\,\sqrt{1-\beta_t}\,\mathbf z_{t-1},\,\beta_t\mathbf I\right)$$

$I$0

$I$1

This approach supports robust adaptation to cross-identity, multi-character cases with diverse environmental and body-shape factors (Yan et al., 9 Jun 2026).

4. Dataset Statistics, Structure, and Annotation

MotionPair-60K comprises 59,376 end-to-end motion-transfer pairs:

Task Type	Count
Single-character animation (SCAIL)	31,895
Single-character animation (Wan-Animate)	13,847
Single-character replacement (MoCha)	9,249
Multi-character replacement (MoCha)	4,385

An additional ~100,000 pose-driven pairs further augment diversity.

• Sampling Ratios: 60% end-to-end animation, 20% end-to-end replacement, 20% pose-driven.
• Resolutions: Videos typically resized to latent grids (e.g., 16 frames × 256×256 px, with I2V backbones defining temporal and spatial resolution: $I$2, $I$3, $I$4).
• Data Modalities: Each sample includes the driving video $I$5, reference image $I$6, ground-truth video $I$7, in-context mask channels (environment and binding slots), and optional pose skeleton/mesh (for pose-driven cases).
• Masks & Channels: One "Environment Switch" channel plus $I$8 "Binding Slot" channels ($I$9 by default), temporally stacked and aligned via the SAM3 segmentor.
• Format: Videos in .mp4 or image (.png/.jpg) format; masks as per-frame, per-channel NumPy arrays (.npy); metadata per sample in .json with modality paths, task mode, character list, and environment source indicator (Yan et al., 9 Jun 2026).

5. Conditioning and Training Scheme

SCAIL-2 leverages in-context mask conditioning and mode-specific RoPE for soft guidance, in addition to textual instructions and visual frames. The model architecture supports concatenated spatial and channel-wise inputs. During training, context is formed by encoding reference images, driving sequences, and noisy target videos, then concatenating mask channels. Prediction and loss computation utilize a UNet-based diffusion process, as outlined in the core code snippets:

$I$1 Bias-Aware DPO is integrated to mitigate synthetic discrepancies, particularly for detailed regions such as hands and fingers, by constructing preference items that guide post-training refinement (Yan et al., 9 Jun 2026).

6. Splits, Use Cases, and Limitations

MotionPair-60K is divided into 80% training, 10% validation, and 10% test splits for end-to-end pairs, with pose-driven data following matching proportions. The dataset sources driving footage from a mixture of internal and public datasets (e.g., HuMo) and reference imagery from broad human-centric corpora. The synthetic loop enforces cross-identity coverage, body-shape variation, background complexity, and multi-character interaction. The license is permissive academic.

Intended use cases include:

• Training end-to-end video diffusion models for character animation
• Research on universal motion transfer, multi-character interactions, and environment weaving
• Fine-grained post-training (e.g., Bias-Aware DPO for extremity articulation)

Known limitations primarily arise from synthetic generator bias (notably for hands and faces), dependence on the quality of generator $I$0 and segmentor (SAM3), and underrepresentation of extreme facial expressions, long sequences, and non-human creatures (Yan et al., 9 Jun 2026).

7. Comparative Analysis and Research Utility

Compared to existing datasets such as TED-Human and AIST-Dance, MotionPair-60K offers explicit cross-identity pairing and heterogeneous task support by design, which real pose-driven datasets lack. When contrasted with Unreal Engine–rendered datasets (e.g., MoCha), MotionPair-60K exhibits greater character diversity and reduced artifacts by combining multiple synthetic generators with reverse-driving to enforce realism. It also subsumes single-task datasets by unifying animation, replacement, and multi-character setups in a consistent, end-to-end format.

MotionPair-60K and SCAIL-2 thus provide unique infrastructure for rapid prototyping and evaluation of controllable video diffusion architectures in universal character animation, supporting rigorous cross-domain validation and development (Yan et al., 9 Jun 2026).