Kling-MotionControl: Holistic Character Animation

• Kling-MotionControl is a unified DiT-based framework that transfers motion from a driving video to a reference image while preserving appearance and identity.
• The system uses specialized representations for body, face, and hands along with adaptive identity-agnostic learning to manage heterogeneous motion cues.
• It accelerates inference by over 10x through a dual-branch sampling and multi-stage distillation pipeline, integrating text responsiveness for detailed animation control.

Kling-MotionControl is a unified DiT-based framework engineered specifically for robust, precise, and expressive holistic character animation. Its core task is image-to-video character animation: given a reference image whose appearance should be preserved and a driving video whose motion should be imitated, it synthesizes a video in which the reference subject performs the motions from the driving video. The system is presented as a high-fidelity, 1080p-capable character-animation model that combines heterogeneous motion representations for body, face, and hands, adaptive identity-agnostic learning for cross-identity retargeting, dedicated identity injection and fusion, a subject library mechanism, text responsiveness through a Prompt Enhancer, and a multi-stage distillation pipeline that boosts inference speed by over 10x (Team et al., 3 Mar 2026).

1. Problem setting and scope

Kling-MotionControl addresses character animation as motion transfer. If $I_{\text{ref}}$ is the reference image and $V_{\text{drv}} = \{f_t\}_{t=1}^T$ is the driving video, the output is a synthesized video $\hat{V} = \{\hat{f}_t\}_{t=1}^T$ such that the appearance and identity in all frames $\hat{f}_t$ match $I_{\text{ref}}$, while the motion of body, face, and hands in $\hat{V}$ matches the motion in $V_{\text{drv}}$ after retargeting across different identities (Team et al., 3 Mar 2026).

The report defines the system as a holistic character-animation model. “Holistic” denotes simultaneous coordination of body, face, and hands: body motion involves global pose, limb motion, large-amplitude movements, and viewpoint; face motion involves head pose, expressions, lip motion, and micro-expressions; hand motion involves finger articulation, hand pose, and hand-object interactions. The technical motivation is that these regions behave differently: body motion is large-scale, structured, and relatively low-frequency; facial motion is dense, subtle, and highly expressive; hand motion is fine-grained, high-frequency, and easily degraded in generation. A single homogeneous motion representation is therefore described as either too coarse for face and hands or too fragile and noisy for large body motion (Team et al., 3 Mar 2026).

The target operating regime is not limited to photorealistic human reenactment. The report emphasizes open-domain generalization across realistic humans, stylized cartoons, anime, stylized artworks, and, to some extent, animals. It also frames robustness in terms of large and fast motions, complex self-occlusion and perspective, and cross-identity transfer such as human-to-child or realistic-to-cartoon retargeting (Team et al., 3 Mar 2026).

2. DiT-based architecture and unified conditioning

The high-level pipeline begins with input processing. Motion signals are extracted from the driving video for body pose, face motion, and hand pose, while appearance and identity features are extracted from the reference image and, optionally, from a subject library of additional images or video of the same subject. User text prompts describing scene, style, camera motion, and related attributes are encoded in parallel. Each of body, face, and hands has a specialized representation and encoder, and the resulting motion embeddings are fused with identity and text conditioning before entering the generative backbone (Team et al., 3 Mar 2026).

The generative backbone is a Diffusion Transformer operating in a latent video space via 3D VAE-style compression. The report follows the standard diffusion formulation and gives the typical objective

$$\mathcal{L}_{\text{diff}} = \mathbb{E}_{x_0, \epsilon, t} \big[ \|\epsilon - \epsilon_\theta(x_t, t, c)\|_2^2 \big],$$

where $x_0$ is the clean video latent, $\epsilon \sim \mathcal{N}(0, I)$ is Gaussian noise, $V_{\text{drv}} = \{f_t\}_{t=1}^T$0 is the noised sample at diffusion step $V_{\text{drv}} = \{f_t\}_{t=1}^T$1, and $V_{\text{drv}} = \{f_t\}_{t=1}^T$2 collects conditioning signals from motion, identity, and text. Within this formulation, Kling-MotionControl keeps the conditioning space unified: body, face, and hand motion are encoded into a common token space that is fed to the DiT together with identity and text features (Team et al., 3 Mar 2026).

The architectural claim is not merely multimodal aggregation but coordinated fusion. The report states that the DiT sees a combined conditioning sequence enabling it to maintain body structure at the macro level while injecting fine facial and hand details at the micro level. This suggests that the model’s central design problem is not only spatiotemporal denoising, but also arbitration among conditioning streams with very different spatial frequencies and temporal semantics (Team et al., 3 Mar 2026).

3. Divide-and-conquer motion strategy and retargeting

The model’s “divide-and-conquer” strategy decomposes motion guidance into three specialized streams. Body motion representation is based on whole-body pose estimation, optimized for large-amplitude motions and viewpoint changes. Face motion representation is tailored to rich and unstructured facial dynamics, including head pose, expressions, and lip motion, while explicitly aiming to filter out identity-specific geometry. Hand motion representation is adapted to dense hand pose, finger joints, and complex gestures, targeting common failure modes such as incorrect finger configurations, broken hands, or artifacts in rapid gestures (Team et al., 3 Mar 2026).

These three streams are integrated through a progressive multi-stage training strategy. The report describes training that first emphasizes body control, then adds face-motion specialization, then integrates hand-motion modeling, and later trains holistic scenes in which body, face, and hands all move together with text and camera controls. The intended outcome is a single model that remains stable under full-body motion while retaining articulatory detail for close-up facial and manual actions (Team et al., 3 Mar 2026).

Cross-identity retargeting is handled through adaptive identity-agnostic motion learning. The report characterizes this as a separation between motion encoders, driven by the source motion video, and identity encoders, driven by the reference appearance input. The motion side emphasizes geometric abstraction and semantic motion modeling rather than absolute pixel-space morphology. A complementary semantic motion layer is described as capturing high-level actions such as “facepalm,” “clapping,” and “waving,” so that the generated animation remains semantically faithful even when source and target bodies differ substantially. Qualitative examples are reported for child-to-adult and realistic-to-cartoon transfer, where motion is retargeted naturally while preserving the reference identity (Team et al., 3 Mar 2026).

The report also attributes part of this robustness to 3D awareness with free-view camera control. Multi-view training and 3D-aware representations are presented as mechanisms that improve alignment under view changes and enable text-driven camera trajectories. A plausible implication is that the model’s motion representation is intended to be viewpoint-robust rather than strictly screen-space (Team et al., 3 Mar 2026).

4. Identity preservation, subject libraries, and text responsiveness

Appearance preservation is treated as a first-class problem. Kling-MotionControl uses dedicated identity encoding and fusion so that face structure, facial features, hairstyle, clothing patterns, and global style remain consistent across long sequences and extreme poses. Identity features are extracted from the reference image and converted into identity tokens, which are then injected into the DiT through dedicated fusion mechanisms so that identity is not overridden by motion structure or text prompts (Team et al., 3 Mar 2026).

To strengthen consistency under viewpoint changes and long-horizon animation, the system supports a subject library. Additional materials for the same character—multi-view images or video clips—are encoded into a richer identity representation. The report explicitly describes the subject library as a memory bank of appearance features, giving better coverage of backside details, improving robustness to viewpoint variation, and supporting stronger consistency through occlusion-reveal cycles and long-duration videos (Team et al., 3 Mar 2026).

Text control is mediated by a Prompt Enhancer. The report presents the Prompt Enhancer as bridging motion control and textual guidance, allowing the model to remain faithful to the driving motion while following user instructions about scene elements, character appearance, style, and camera movements such as pan, zoom, and orbit. It further states that Kling-MotionControl supports “native text descriptions” for camera control and general text responsiveness. The report uses the standard classifier-free guidance form

$V_{\text{drv}} = \{f_t\}_{t=1}^T$3

with $V_{\text{drv}} = \{f_t\}_{t=1}^T$4 including motion, identity, and text conditioning, and later distills this conditional behavior into the student model so that runtime does not require repeated CFG passes (Team et al., 3 Mar 2026).

The control interface is therefore layered. Driving video specifies motion, identity inputs specify appearance, and text refines scene, style, and camera. The report also notes that the degree of text influence can be tuned analogously to CFG scale. This suggests a conditioning hierarchy in which motion remains primary for animation, while text modulates cinematic and stylistic dimensions (Team et al., 3 Mar 2026).

5. Training pipeline, acceleration, and human-preference evaluation

The training pipeline is described at a high level rather than through exhaustive hyperparameters. The dataset is characterized as massive and curated across realistic humans, stylized cartoons and anime, diverse motion dynamics, and varied camera and scene conditions. Preprocessing includes whole-body, hand-pose, and face-motion extraction; multi-dimensional filtering by video quality score, motion dynamics, and subject consistency; supplementary rendered data; high-speed camera footage; and fine-grained annotation of actions, motion categories, micro-expressions, human-object interactions, and camera moves (Team et al., 3 Mar 2026).

For each training sample, the target video $V_{\text{drv}} = \{f_t\}_{t=1}^T$5 is encoded into a latent $V_{\text{drv}} = \{f_t\}_{t=1}^T$6, a reference frame $V_{\text{drv}} = \{f_t\}_{t=1}^T$7 and a driving video $V_{\text{drv}} = \{f_t\}_{t=1}^T$8 are selected, motion and identity features are encoded into conditioning tokens, and the DiT predicts $V_{\text{drv}} = \{f_t\}_{t=1}^T$9 under the diffusion loss. The report also explicitly states that additional objectives are implicitly involved, including reconstruction and consistency, identity preservation, temporal coherence, and semantic alignment. It presents the combined loss only conceptually: $\hat{V} = \{\hat{f}_t\}_{t=1}^T$0 while noting that exact coefficients and forms are not specified in the report (Team et al., 3 Mar 2026).

Acceleration is a major engineering component. The teacher DiT is trained with a large number of diffusion steps and supports multi-conditional classifier-free guidance for motion, appearance, and text. To avoid the cost of naïve multi-branch CFG, the report introduces an efficient dual-branch sampling strategy with a single conditional branch and a single unconditional branch, yielding multi-conditional guidance with two forward passes per step. It then distills this teacher into a few-step student and folds the effect of CFG into the student’s weights so that inference uses only one forward pass per step. The stated result is end-to-end acceleration exceeding 10x while preserving model performance (Team et al., 3 Mar 2026).

Evaluation is centered on a dedicated benchmark of 150 high-quality test cases, each pairing a reference image with a driving video from distinct subjects. Human participants perform Good/Same/Bad pairwise comparisons against Dreamina, Runway Act-Two, and Wan-Animate. The report defines the main metric as

$\hat{V} = \{\hat{f}_t\}_{t=1}^T$1

with higher values indicating that Kling-MotionControl is judged better or not worse than the baseline more often than worse (Team et al., 3 Mar 2026).

The reported GSB ratios are explicitly favorable. Against Dreamina, the ratios are 3.44 overall, 3.33 for Visual Quality, 1.92 for Dynamic Quality, 1.56 for Identity Preservation, 1.05 for Motion Accuracy, and 1.20 for Expression Accuracy. Against Runway Act-Two, they are 16.25 overall, 8.00 for Visual Quality, 4.64 for Dynamic Quality, 2.95 for Identity Preservation, 3.32 for Motion Accuracy, and 4.50 for Expression Accuracy. Against Wan-Animate, they are 4.00 overall, 6.43 for Visual Quality, 1.77 for Dynamic Quality, 3.07 for Identity Preservation, 1.34 for Motion Accuracy, and 1.16 for Expression Accuracy (Team et al., 3 Mar 2026).

6. Relation to adjacent motion-control research

Kling-MotionControl belongs to a broader research landscape of DiT-based video generation and editing systems that seek stronger motion control without sacrificing visual fidelity. SAMA, for example, factorizes instruction-guided video editing into Semantic Anchoring and Motion Alignment inside a single Wan2.1-T2V-14B backbone, and explicitly states that it achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems such as Kling-Omni (Zhang et al., 19 Mar 2026). This is relevant because it isolates a design principle—factorizing semantic planning from motion modeling—that is conceptually adjacent to Kling-MotionControl’s body/face/hand decomposition, even though the task is instruction-guided editing rather than character animation.

STIV provides another point of comparison. It presents a simple scalable recipe for text-image-conditioned video generation based on frame replacement inside a DiT and reports that an 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing CogVideoX-5B, Pika, Kling, and Gen-3, while the same-sized model reaches 90.1 on VBench I2V at 512 resolution (Lin et al., 2024). DOLLAR, by contrast, focuses on few-step sampling and latent reward optimization; its 4-step student reaches 82.57 on VBench and slightly surpasses Kling’s 81.85 while enabling one-step acceleration of up to 278.6 times (Ding et al., 2024). Together, these results suggest that the Kling family occupies a competitive but contested position within the current DiT-based video ecosystem.

A related line of work studies image-mediated and in-frame motion control in Kling-like systems. “In-Video Instructions” reports that Kling 2.5 can follow text and arrows embedded directly in the first frame under a fixed prompt, “Follow the instructions step by step,” and that this in-frame conditioning improves Dynamic Degree on Kling-2.5 from 0.4218 with a Text Prompt to 0.5625 with In-Video Instructions, while multi-object instruction success rates reach 20.8% for “A (Back up),” 58.3% for “B (Turn right),” and 95.8% for “C (Stop),” compared with 8.3%, 29.2%, and 58.3% for prompt-only control (Fang et al., 24 Nov 2025). This literature does not describe the dedicated Kling-MotionControl system itself, but it shows that Kling-class image-to-video models expose a strong zero-shot visual control channel.

7. Safety, failure modes, and dual-use implications

The technical report’s impact statement identifies deepfake-like risks arising from precise control over body dynamics and facial reenactment, faithful identity preservation, and the possibility of unauthorized animation of individuals. It recommends safeguards such as content filtering and watermarking or other authenticity indicators, and explicitly notes the need for legal and ethical frameworks to guide deployment (Team et al., 3 Mar 2026).

Adjacent safety research shows that the same control affordances can be exploited adversarially. “VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models” studies Kling-v2.5-turbo as one of four commercial image-to-video systems and reports that Kling understands typographic instructions in the image, uses bounding boxes and arrows to determine spatial focus and motion direction, and tends to obey these visual instructions even when the accompanying text prompt is innocuous or safety-aligned and even when a system prompt tells it to ignore the overlaid instructions. On COCO-I2VSafetyBench, VII reaches Avg ASR 81.5% under VBench++ and 86.5% under a VLM-based protocol on Kling-v2.5-turbo, both with RR 1.0%; on ConceptRisk, it reaches Avg ASR 63.5% and 68.0%, both with RR 2.0% (Zheng et al., 24 Feb 2026). In that paper, “Kling-MotionControl” also appears as a broader label for Kling’s image-encoded motion-and-scene steering capability rather than solely the dedicated character-animation system.

Text-only jailbreaking has also been reported for Kling-family models. “T2V-OptJail” treats Kling 1.0 as a black-box text-to-video model and reports average ASR 34.7% as assessed by GPT-4, average ASR 33.0% as assessed by humans, and semantic similarity 0.261 for optimized adversarial prompts, improving over the T2VSafetyBench baseline on Kling (Liu et al., 10 May 2025). This suggests that the control surface is dual-use across both image and text channels.

Failure modes in the benign-control literature are more prosaic but structurally related. “In-Video Instructions” notes persistent visual markers, partial erasure of arrows or text, ambiguity when instructions are too close together, and conflicts between motion instructions and learned physical priors—for example, relatively low success for the “Back up” car instruction because the car is physically blocked by another behind it (Fang et al., 24 Nov 2025). The Kling-MotionControl report itself does not dwell on explicit failure cases, but it acknowledges that extremely rare or heavily out-of-distribution actions, very unusual viewpoints, rare stylizations, and highly intricate hand-object interactions may remain challenging (Team et al., 3 Mar 2026).

Taken together, the literature presents Kling-MotionControl as a technically sophisticated holistic animation system whose principal innovations—heterogeneous motion representations, identity-retargeting, strong image-conditioned control, and distilled responsiveness to text—also define its main governance challenges. The same mechanisms that support high-quality, controllable, and lifelike character animation support a broad spectrum of visual command channels, some benign and some adversarial (Team et al., 3 Mar 2026).