Video-to-Video Motion Personalization

• Video-to-video motion personalization is the process of transferring motion from a reference video to a target video while clearly separating dynamic movement from static appearance.
• It utilizes diverse methods such as adapter-based diffusion, guidance-driven control, and pose-first retargeting to enhance temporal coherence and semantic fidelity.
• Applications include text-to-video synthesis, camera motion transfer, and localized editing, with ongoing challenges in long-term consistency and multi-entity interactions.

Video-to-video motion personalization denotes the extraction of motion from a reference video and its reuse in a newly synthesized video whose appearance, subject, scene, or edit target differs from the source. Across the recent literature, the problem appears in several closely related forms: one-shot or few-shot motion customization for text-to-video and image-to-video diffusion models, camera-motion transfer, localized trajectory-driven video editing, multi-subject interactive motion generation, and human motion-style transfer from unconstrained videos (Jeong et al., 2023, Wei et al., 2023, Wang et al., 2024, Guhan et al., 13 Apr 2025, Tan et al., 30 Jun 2025, Zhang et al., 5 Jan 2026). The central technical difficulty is not merely reproducing motion, but separating motion from appearance strongly enough that the transferred dynamics remain faithful while the generated video retains semantic controllability, subject identity, spatial coherence, and temporal consistency.

1. Historical development and problem scope

Before diffusion-based personalization, video-to-video motion transfer was dominated by pose-mediated retargeting. TransMoMo models motion, structure, and view-angle as disentangled latent factors from 2D joint trajectories, then recombines source motion with target structure and view in an unsupervised auto-encoding framework (Yang et al., 2020). “Do as we do” performs multiple-person video-to-video transfer by converting source and target videos into temporally consistent, color-coded stick-figure representations, then rendering frames with a three-stage conditional GAN pipeline that preserves actor identity, floor contact, and relative spatial arrangement (Cormier et al., 2021). In these systems, motion is represented explicitly as skeletal or pose information.

Diffusion-era formulations substantially broaden the task. VMC describes the challenge as twofold: accurately reproducing motion from a target video and creating diverse visual variations, while noting that straightforward extensions of static image customization methods to video often lead to intricate entanglements of appearance and motion data (Jeong et al., 2023). DreamVideo decouples the problem into subject learning from a few static images and motion learning from a few videos of target motion (Wei et al., 2023). Separate Motion from Appearance formalizes motion customization as adapting a diffusion model with a small set of reference clips illustrating a novel motion concept while preserving the ability to generate diverse appearances (Liu et al., 28 Jan 2025). PersonaAnimator goes further and explicitly introduces “Video-to-Video Motion Personalization” as a task in which a content video provides basic human motion and a style video provides a person’s unique motion characteristics, with the goal of synthesizing a video that preserves the skeleton “content” trajectory while infusing the “style” dynamics of the reference (Qian et al., 27 Aug 2025).

This suggests a continuum rather than a single task definition. At one end are retargeting systems that preserve explicit body structure; at the other are generative systems that personalize latent motion priors without explicit pose supervision. Between them lie camera-motion transfer, localized motion editing, and multi-entity composition (Guhan et al., 13 Apr 2025, Mou et al., 2024, Zhang et al., 8 Jul 2025).

2. Architectural patterns and motion representations

A useful synthesis distinguishes three recurrent design patterns in the literature.

Design pattern	Motion representation	Representative papers
Adapter- or embedding-based diffusion customization	temporal attention updates, motion embeddings, motion words, motion residuals	VMC, DreamVideo, Motion Inversion, SAVE, Set-and-Sequence, SynMotion, MoTrans
Guidance-driven control at sampling time	homographies, trajectory maps, attention-derived motion fields, two-branch guidance	CamMimic, ReVideo, MotionAdapter, VideoMage, Tora2
Pose- or skeleton-first retargeting	2D joints, stick figures, style-conditioned skeleton sequences	TransMoMo, Do as we do, PersonaAnimator

Within UNet-based video diffusion, motion is often isolated in temporal modules and appearance in spatial modules. DreamVideo inserts an Identity Adapter in spatial cross-attention layers and a Motion Adapter in all layers of the temporal transformer, while keeping the original UNet and text and CLIP encoders frozen (Wei et al., 2023). Motion Inversion instead introduces explicit Motion Query-Key Embeddings to modulate the temporal attention map and Motion Value Embeddings to modulate attention values across frames, thereby leaving spatial attention and convolution untouched (Wang et al., 2024). SAVE represents motion as a learned pseudo-token $S_{\mot}$ with a frame-varying textual embedding, coupled to a pseudo optical flow mined from spatio-temporal attention maps (Song et al., 2023).

In DiT-based video generation, the separation between spatial and temporal factors is less explicit, so several methods impose one. Set-and-Sequence learns an identity LoRA basis from an unordered set of frames and then adds Motion Residuals on the full video sequence, yielding

$W' = W + A_1 B_1 + A_1 B_2,$

where $A_1,B_1$ encode appearance and $B_2$ encodes motion residuals (Abdal et al., 20 Feb 2025). SynMotion augments each attention layer with low-rank motion adapters in the $Q,K,V$ projections and couples them with a dual-embedding semantic mechanism that disentangles subject and motion embeddings before fusing them with an Embedding Refiner (Tan et al., 30 Jun 2025). Tora2 uses a Decoupled Personalization Extractor to form per-entity personalization embeddings and a gated self-attention mechanism to bind those embeddings with trajectory and text tokens (Zhang et al., 8 Jul 2025).

Other systems represent motion more explicitly. CamMimic estimates frame-to-frame homographies from a reference video and uses them both as a camera-motion descriptor and as weak supervision during inference (Guhan et al., 13 Apr 2025). ReVideo uses frame-wise trajectory maps $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$ that encode user-specified displacements inside an edited region (Mou et al., 2024). MotionAdapter derives motion fields directly from cross-frame attention in DiT full-attention blocks, then customizes those fields via DINO-guided correspondences between reference and target content (Zhang et al., 5 Jan 2026).

3. Disentangling motion from appearance

Motion-appearance disentanglement is the defining technical theme of the area. The literature repeatedly treats appearance leakage as the main failure mode of naïve motion transfer.

VMC addresses the problem by adapting temporal attention layers with a one-shot tuning approach and by introducing a motion distillation objective using residual vectors between consecutive frames as a motion reference; the diffusion process is designed to preserve low-frequency motion trajectories while mitigating high-frequency motion-unrelated noise in image space (Jeong et al., 2023). DreamVideo reduces appearance contamination in motion learning by feeding the Motion Adapter a frozen CLIP image embedding from one frame as appearance guidance, so that the temporal module focuses on motion rather than relearning subject identity (Wei et al., 2023). Motion Inversion makes the same separation more explicit: its Motion Query-Key Embedding has no spatial span and therefore cannot carry spatial detail, while its Motion Value Embedding is debiased at inference time by subtracting the per-frame mean over channels (Wang et al., 2024).

Several papers operationalize disentanglement through restricted parameterization. Separate Motion from Appearance assumes that pretrained Value embeddings in temporal attention are sufficient as reusable components for generating a new motion, and therefore adapts only the Key projection $W_K$ with LoRA while freezing $W_Q$ and $W_V$; it then reroutes UNet skip connections from temporal outputs to spatial outputs through an Appearance Highway and introduces Phased LoRA Integration so that later denoising steps revert to the vanilla UNet (Liu et al., 28 Jan 2025). VideoMage likewise inserts a motion LoRA only in temporal blocks and uses negative classifier-free guidance with an appearance prompt to erase appearance information from the motion target during training (Huang et al., 27 Mar 2025). CamMimic learns spatial and temporal LoRAs in two stages and enforces orthogonality between spatial- and temporal-LoRA weights,

$$\mathcal{L}_{\text{ortho}} = \langle W_{\text{spatial\_LoRA}}, W_{\text{temporal\_LoRA}} \rangle,$$

to preserve camera-motion patterns while blending in the user image features (Guhan et al., 13 Apr 2025).

Semantic disentanglement has also become prominent. MoTrans employs an MLLM-based recaptioner to expand the prompt toward appearance details, an appearance injection module that feeds a video-frame embedding into temporal layers, and a motion-specific embedding built from the verb token and a mean-pooled video feature (Li et al., 2024). SynMotion explicitly partitions prompt embeddings into subject and motion components, learns residuals $W' = W + A_1 B_1 + A_1 B_2,$0 and $W' = W + A_1 B_1 + A_1 B_2,$1, and alternates optimization on user-provided exemplar videos and the Subject Prior Video dataset so that motion specificity is improved without collapsing subject generalization (Tan et al., 30 Jun 2025). The repeated appearance of orthogonality losses, frozen appearance priors, negative guidance, skip-path rerouting, and dual embedding schemes indicates that temporal finetuning alone is generally insufficient for robust motion transfer.

4. Conditioning, guidance, and sampling-time control

A second major axis of development concerns how motion is enforced at inference time. Early motion-personalization methods often relied primarily on learned adapters or embeddings. More recent systems increasingly add explicit guidance during denoising.

CamMimic is exemplary for camera-motion transfer. After multi-concept finetuning, it extracts homographies

$W' = W + A_1 B_1 + A_1 B_2,$2

warps the first user frame forward to form a weak pseudo video, and at each sampling step nudges the latent $W' = W + A_1 B_1 + A_1 B_2,$3 toward the weak latent $W' = W + A_1 B_1 + A_1 B_2,$4 with a gradient step

$W' = W + A_1 B_1 + A_1 B_2,$5

This homography-guided inference is paired with CameraScore,

$W' = W + A_1 B_1 + A_1 B_2,$6

which measures camera-motion similarity across unrelated scenes (Guhan et al., 13 Apr 2025).

ReVideo addresses localized motion editing rather than holistic transfer. Its input includes a user-modified first frame, a binary mask, and frame-wise trajectory maps. The method uses a three-stage training curriculum—Motion-Prior Training, Decoupling Training, and Deblocking Training—and a Spatiotemporal Adaptive Fusion Module that blends content and motion features as

$W' = W + A_1 B_1 + A_1 B_2,$7

with $W' = W + A_1 B_1 + A_1 B_2,$8 conditioned on the edit mask and diffusion timestep (Mou et al., 2024). This design makes motion control sparse, local, and time-dependent.

MotionAdapter generalizes guidance to DiT attention. It reads cross-frame attention blocks at a selected noise step and layer to estimate raw motion fields, customizes those fields via Lang-SAM foreground segmentation and DINO-v2 plus Hungarian matching, smooths the result, and then applies latent correction during the first $W' = W + A_1 B_1 + A_1 B_2,$9 of denoising steps so that the target motion field matches the customized reference motion (Zhang et al., 5 Jan 2026). VideoMage uses a different sampling-time strategy: a subject branch and a motion branch denoise in parallel, spatial cross-attention and temporal self-attention maps are aligned through collaborative guidance in early steps, and the predicted noises are fused into a final update (Huang et al., 27 Mar 2025). Tora2 also places significant emphasis on the order and site of conditioning injection: motion tokens are injected via adaptive LayerNorm and personalization tokens via an extra cross-attention layer, with the reported “Motion→Personalization” order outperforming the reverse in trajectory error (Zhang et al., 8 Jul 2025).

These methods indicate a shift from purely learned motion priors toward hybrid systems in which training-time personalization is supplemented by explicit geometric or attention-derived control signals during sampling.

5. Evaluation protocols and empirical status

Evaluation remains heterogeneous. Different papers target different subproblems—general motion transfer, camera-motion transfer, localized editing, dynamic concept personalization, or human motion style—and therefore report different metrics. This heterogeneity is itself a salient fact of the field.

SynMotion introduces MotionBench, curated with 16 challenging motion categories and 6–10 real-world exemplar videos per motion, and evaluates both T2V and I2V settings with Motion Accuracy, Motion Consistency or MotionSmooth, Subject Accuracy, Subject Consistency, Imaging Quality, Dynamic Degree, Background Consistency, TempFlick, and Aesthetic (Tan et al., 30 Jun 2025). On MotionBench T2V, SynMotion reports $A_1,B_1$0 MotionAcc, $A_1,B_1$1 MotConsist, $A_1,B_1$2 SubjAcc, $A_1,B_1$3 SubjConsist, $A_1,B_1$4 ImgQual, $A_1,B_1$5 DynDegree, and $A_1,B_1$6 BkgConsist; on I2V it reports $A_1,B_1$7 MotionAcc, $A_1,B_1$8 MotionSmooth, $A_1,B_1$9 TempFlick, $B_2$0 I2VSubjAcc, $B_2$1 SubjConsist, $B_2$2 BkgConsist, and $B_2$3 Aesthetic (Tan et al., 30 Jun 2025).

DreamVideo evaluates both motion-only and joint subject-plus-motion personalization. In motion-only personalization it reports CLIP-T $B_2$4, Temporal Consistency $B_2$5, and $B_2$6 parameters. In joint personalization it reports CLIP-T $B_2$7, CLIP-I $B_2$8, DINO-I $B_2$9, Temporal Consistency $Q,K,V$0, and $Q,K,V$1 parameters (Wei et al., 2023). Separate Motion from Appearance reports one-shot gains over MotionDirector, with Text Alignment $Q,K,V$2, Temporal Consistency $Q,K,V$3, Aesthetic $Q,K,V$4, ViCLIP $Q,K,V$5, and VBench $Q,K,V$6; in the few-shot setting it reports Text Alignment $Q,K,V$7, Temporal Consistency $Q,K,V$8, Aesthetic $Q,K,V$9, ViCLIP $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$0, and VBench $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$1 (Liu et al., 28 Jan 2025).

For camera motion, CamMimic evaluates on 680 reference-video and user-image pairs and reports CameraScore $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$2–$c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$3 lower than all baselines, DINOv2 similarity of approximately $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$4, VideoCLIP mean score $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$5, and user preferences of $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$6 for scene preservation and $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$7 for motion transfer (Guhan et al., 13 Apr 2025). For localized editing, ReVideo reports PSNR $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$8, Text-Align $c_{\mot} \in \mathbb{R}^{N \times 2 \times H \times W}$9, Consistency $W_K$0, Human Overall $W_K$1, and Editing-Target $W_K$2 (Mou et al., 2024). MotionAdapter, on a 150-pair subset of DAVIS, reports CLIPScore $W_K$3, Motion Fidelity $W_K$4, FVD $W_K$5, KVD $W_K$6, and more than $W_K$7 user preference; the paper explicitly notes that its Motion Fidelity score under-reports gains on large semantic-gap scenarios (Zhang et al., 5 Jan 2026).

The cumulative picture is consistent even when metrics differ: better motion transfer is no longer judged only by temporal smoothness or text alignment, but by a multi-objective balance among motion fidelity, subject fidelity, scene preservation, appearance diversity, background stability, and edit locality.

6. Applications, failure modes, and open directions

The application space now spans several distinct operating regimes. DreamVideo combines a new subject from 3–5 photos with a target motion pattern from one or more 32-frame videos (Wei et al., 2023). ReVideo supports locally changing video content while keeping motion constant, keeping content unchanged and customizing new motion trajectories, modifying both content and motion trajectories, and extending these operations to multi-area editing without specific training (Mou et al., 2024). VideoMage addresses multiple subjects and their interactive motions through subject- and motion-LoRAs plus a spatial-temporal composition scheme (Huang et al., 27 Mar 2025). CamMimic specializes in holistic camera motion transfer from a reference video onto an arbitrary user-provided still image (Guhan et al., 13 Apr 2025). PersonaAnimator addresses personalized human motion style transfer directly from unconstrained videos and adds Physics-aware Motion Style Regularization to discourage bone stretching and joint collapse (Qian et al., 27 Aug 2025).

Failure modes remain well documented. DreamVideo notes multi-object motion fusion failures, coarse rather than frame-by-frame alignment for extremely stylized source motions, and dependence on the frozen UNet’s spatiotemporal prior for unseen interactions such as “wolf riding a bicycle” (Wei et al., 2023). ReVideo reports that very long videos require sliding-window inference and may accumulate drift (Mou et al., 2024). MoTrans states that objects without limbs cannot execute limb-centric motions such as “running,” and that current focus remains on 2–3 s clips (Li et al., 2024). PersonaAnimator depends on 2D pose estimation quality and currently operates with a fixed set of 120 styles (Qian et al., 27 Aug 2025). Separate Motion from Appearance identifies multi-object and extremely long motions as persistent sources of entanglement (Liu et al., 28 Jan 2025).

Several research directions follow directly from these limitations. One is longer-horizon personalization: many current systems remain optimized for short clips or early-denoising control windows (Li et al., 2024, Zhang et al., 5 Jan 2026). A second is broader generalization across appearance and semantics: methods such as MotionAdapter and CamMimic explicitly target large semantic or scene gaps, but their very existence indicates that transfer across strong content mismatch is still nontrivial (Guhan et al., 13 Apr 2025, Zhang et al., 5 Jan 2026). A third is stronger physical plausibility and structure preservation, especially for personalized human motion, where PersonaAnimator’s stability and connectivity losses suggest a move beyond purely perceptual objectives (Qian et al., 27 Aug 2025). A fourth is multi-entity binding and interaction, addressed by VideoMage and Tora2 but not yet standardized as a general benchmarked setting (Huang et al., 27 Mar 2025, Zhang et al., 8 Jul 2025).

Video-to-video motion personalization has therefore evolved from pose retargeting into a broader family of generative control problems. The common objective is stable across formulations: retain target appearance or content, import reference dynamics, and do so without collapsing semantic controllability or temporal coherence. What differs is the representation of motion—pose, attention, homography, trajectory, token, adapter, or motion field—and the location at which the transfer is enforced: in training objectives, in architectural factorization, or directly in the denoising trajectory itself.