MotionShot: Text-to-Video Motion Transfer

• MotionShot is a training-free framework for text-to-video generation that retargets motion from a reference object onto a text-specified target through fine-grained correspondence parsing.
• It combines high-level semantic feature matching, low-level morphological retargeting via thin-plate-spline warping, and attention-based temporal guidance for coherent motion transfer.
• The method achieves strong performance on CLIP metrics and user studies, effectively handling significant appearance and structure disparities.

Searching arXiv for the MotionShot paper and closely related components mentioned in the provided data. MotionShot is a training-free framework for text-to-video generation that retargets the motion of an arbitrary reference object onto a text-specified target object through fine-grained reference-target correspondence parsing. It is designed for the setting in which existing text-to-video methods struggle to transfer motion smoothly when the reference and target differ substantially in appearance or structure. The method combines high-level semantic feature matching, low-level morphological alignment via shape retargeting, and attention-based temporal motion guidance, with the stated goal of achieving high-fidelity motion transfer while preserving coherence in appearance (Liu et al., 22 Jul 2025).

1. Conceptual scope and problem formulation

The central problem addressed by MotionShot is cross-object motion transfer under significant appearance and structure disparities. The framework takes as input reference video frames $\{I_{\mathrm{ref}}^t\}_{t=0\ldots F-1}$ and a text prompt describing the target object, then generates a video in which the target object follows the reference motion while remaining consistent with the textual specification (Liu et al., 22 Jul 2025).

The method is explicitly organized into three stages: high-level semantic feature matching, low-level morphological shape retargeting, and attention-based temporal motion guidance. This decomposition reflects a two-level alignment strategy. First, semantic correspondence is used to align object regions at a high level. Second, morphological retargeting adjusts the reference motion into a target-compatible shape space. Finally, temporal attention is used to inject motion into a text-to-video generator (Liu et al., 22 Jul 2025).

A common misconception is that motion transfer across arbitrary objects can be achieved by directly copying temporal cues from a reference sequence. MotionShot instead assumes that direct transfer is insufficient when the objects differ in geometry or topology. The reported ablations support this position: direct attention without warping produces distorted output, and simple resizing matches scale but not articulation, leading to leg and topology errors (Liu et al., 22 Jul 2025).

2. High-level semantic correspondence parsing

MotionShot begins by synthesizing a single “fake” image $I_{\mathrm{fake}}$ using a text-to-image model, specified as StableDiffusion+ControlNet-seg. The generated image follows the text prompt while sharing the coarse initial pose of $I_{\mathrm{ref}}^0$ through a weak segmentation hint. This intermediate image serves as a proxy target for correspondence estimation rather than as the final generation result (Liu et al., 22 Jul 2025).

Structure-aware keypoint sampling is then performed on the reference frame. Both $I_{\mathrm{ref}}^0$ and $I_{\mathrm{fake}}$ are segmented with SAM, producing binary masks $M_{\mathrm{ref}}$ and $M_{\mathrm{fake}}$. MotionShot samples $m$ keypoints $$K_{\mathrm{ref}}^0=\{p_j\in\mathbb{R}^2\}_{j=1\ldots m}$$ on the reference by combining contour-uniform sampling at interval $d$ with Poisson-disk sampling in the interior. This hybrid sampling procedure is intended to cover both boundary structure and interior support (Liu et al., 22 Jul 2025).

Feature extraction combines low-level diffusion features $I_{\mathrm{fake}}$0 and high-level DINO features $I_{\mathrm{fake}}$1. After PCA-based dimensionality reduction, upsampling to a common resolution, $I_{\mathrm{fake}}$2 normalization, and concatenation, MotionShot obtains feature descriptors $I_{\mathrm{fake}}$3. Pixelwise similarity is defined as

$I_{\mathrm{fake}}$4

For each sampled reference keypoint $I_{\mathrm{fake}}$5, the corresponding target keypoint is selected by

$I_{\mathrm{fake}}$6

This produces the initial target keypoints $I_{\mathrm{fake}}$7 (Liu et al., 22 Jul 2025).

The ablation results clarify why the feature fusion is integral rather than incidental. SD-only features provide good spatial detail but are noisy in texture-less regions; DINO-only features provide strong semantics but miss fine edges; the X-Pose detector is described as too sparse and non-uniform. The fused SD+DINO representation is reported to combine spatial precision and semantic robustness, yielding the most accurate region correspondences (Liu et al., 22 Jul 2025).

3. Morphological retargeting and thin-plate-spline warping

After semantic matching, MotionShot constructs a temporally evolving correspondence field. Each reference keypoint $I_{\mathrm{fake}}$8 is tracked through the reference video using CoTracker3, yielding a sequence $I_{\mathrm{fake}}$9. For each frame $I_{\mathrm{ref}}^0$0, the method estimates a global motion, specifically a center shift $I_{\mathrm{ref}}^0$1 and a rotation $I_{\mathrm{ref}}^0$2, by fitting an ellipse to the set $I_{\mathrm{ref}}^0$3. The same similarity transform is applied to the initial target keypoints $I_{\mathrm{ref}}^0$4, producing global target keypoints

$I_{\mathrm{ref}}^0$5

The formulation also allows optional local refinement in polar coordinates to capture stretching (Liu et al., 22 Jul 2025).

The framewise retargeting is then realized through thin-plate-spline warping. For each frame $I_{\mathrm{ref}}^0$6, MotionShot solves for a warp $I_{\mathrm{ref}}^0$7 such that

$I_{\mathrm{ref}}^0$8

while minimizing the bending energy

$I_{\mathrm{ref}}^0$9

The closed-form TPS solution is

$I_{\mathrm{ref}}^0$0

with

$I_{\mathrm{ref}}^0$1

Each reference frame is then warped as $I_{\mathrm{ref}}^0$2, yielding a motion-aligned video with the target’s shape (Liu et al., 22 Jul 2025).

This retargeting stage is the principal mechanism by which MotionShot separates motion trajectory transfer from object morphology preservation. The reported ablations show that omitting warping causes object-shape mismatches and distorted output, while simple resizing preserves scale only coarsely and fails on articulated parts. By contrast, keypoint-TPS is reported to preserve both motion trajectories and target morphology, producing coherent, artifact-free videos (Liu et al., 22 Jul 2025).

4. Temporal attention guidance within diffusion sampling

MotionShot encodes motion through temporal attention rather than through explicit retraining of a video generator. The warped reference sequence $I_{\mathrm{ref}}^0$3 undergoes forward DDIM inversion at a single diffusion timestep $I_{\mathrm{ref}}^0$4, exposing self-attention

$I_{\mathrm{ref}}^0$5

where $I_{\mathrm{ref}}^0$6 denotes the attention weight from frame $I_{\mathrm{ref}}^0$7 to frame $I_{\mathrm{ref}}^0$8 at spatial index $I_{\mathrm{ref}}^0$9 (Liu et al., 22 Jul 2025).

This attention tensor is sparsified by keeping the top-$I_{\mathrm{fake}}$0 temporal weights per frame, producing a binary mask $I_{\mathrm{fake}}$1. During AnimateDiff sampling, MotionShot defines the energy

$I_{\mathrm{fake}}$2

and uses the guided noise estimate

$I_{\mathrm{fake}}$3

The sampling step is written as

$I_{\mathrm{fake}}$4

According to the formulation, this forces the generated video’s temporal attention to match the warped reference’s motion while its appearance follows the text prompt (Liu et al., 22 Jul 2025).

The significance of this design lies in how it externalizes motion control. Motion is not represented as explicit kinematics, nor is it learned through additional finetuning. Instead, temporal self-attention extracted from a morphology-aligned reference is used as a guidance signal. This suggests a view of motion transfer in which attention structure serves as the primary transport medium once correspondence and shape compatibility have been established.

5. Experimental setting and reported results

The evaluation uses 40 reference videos drawn from DAVIS and online sources, partitioned into 10 human, 20 animal, and 10 other motions. The reported metrics are CLIP-based “Text Alignment” and “Temporal Consistency,” together with a user study with $I_{\mathrm{fake}}$5 in which Motion Preservation, Appearance Diversity, Text Alignment, and Temporal Consistency are scored on a 1–5 scale (Liu et al., 22 Jul 2025).

The quantitative comparison includes VideoComposer, Gen-1, VMC, Tune-A-Video, Control-A-Video, MotionClone, and MotionShot. MotionShot reports a CLIP TextAlign score of 26.95 and a CLIP TempCons score of 97.81. In the user study, it reports 4.95 for Motion Preservation, 4.95 for Appearance Diversity, 4.94 for Text Alignment, and 4.90 for Temporal Consistency. The listed baselines have lower scores on all four user-study criteria, and lower CLIP scores on the two reported automatic metrics (Liu et al., 22 Jul 2025).

Qualitatively, MotionShot is reported to outperform the baselines when the reference and target exhibit large appearance or structure gaps. The examples given are horse→giraffe and anime boy→Winnie Bear. These cases are presented as demonstrations that the framework can transfer motion coherently even when reference and target objects are not close visual analogues (Liu et al., 22 Jul 2025).

A potential misunderstanding is that the method’s main benefit is text alignment alone. The reported results do not support that narrower reading. The gains are simultaneously reported in text alignment, temporal consistency, motion preservation, and appearance diversity, which is consistent with the claim that MotionShot addresses both semantic and morphological aspects of transfer rather than a single downstream metric (Liu et al., 22 Jul 2025).

6. Ablation findings, operating regime, and limitations

The ablation on keypoint count tests $I_{\mathrm{fake}}$6. The reported outcomes are specific: $I_{\mathrm{fake}}$7 yields under-constrained TPS and poor shape match; $I_{\mathrm{fake}}$8 leads to over-fitting and unstable warps; $I_{\mathrm{fake}}$9 provides a balance of flexibility and stability and gives the best visual consistency (Liu et al., 22 Jul 2025). This indicates that the TPS stage has a nontrivial bias-variance tradeoff in correspondence density.

The feature ablation separates the semantic-matching stage into SD-only, DINO-only, X-Pose, and fused SD+DINO variants. The conclusions are likewise explicit: SD-only is detailed but noisy in texture-less regions, DINO-only is semantically strong but weak on fine edges, X-Pose is too sparse and non-uniform, and the fusion gives the most accurate region correspondences (Liu et al., 22 Jul 2025). The paper’s two-level alignment strategy therefore depends materially on feature complementarity rather than on a single universal descriptor.

The shape-retargeting ablation compares no warp, simple resizing, and keypoint-TPS. No warp causes object-shape mismatches and distorted output; simple resizing induces articulation errors; keypoint-TPS preserves both motion trajectories and target morphology (Liu et al., 22 Jul 2025). This directly supports the broader claim that semantic matching alone is not sufficient for cross-object motion transfer.

The principal stated limitation is that MotionShot assumes some semantic similarity between reference and target. In fully unrelated pairs, correspondence breaks down (Liu et al., 22 Jul 2025). A plausible implication is that the method’s applicability is broad but not unconstrained: “arbitrary objects” is operationalized through a correspondence pipeline that still requires enough shared semantics for matching and retargeting to remain meaningful.

7. Position within text-to-video motion transfer

MotionShot is characterized by the conjunction of three properties: it is training-free, it handles significant appearance and structure disparities, and it performs motion transfer through two-level alignment plus attention-guided diffusion (Liu et al., 22 Jul 2025). Within text-to-video generation, this places it in a category distinct from methods that rely primarily on direct appearance conditioning, monolithic motion control, or retraining-based specialization.

Its pipeline can be summarized compactly. A fake target image is first generated under the text prompt and coarse pose constraint. Reference keypoints are sampled and semantically matched to target locations using fused SD and DINO features. These correspondences are temporally propagated with CoTracker3, converted into target-compatible trajectories through global motion estimation and TPS warping, and finally encoded as temporal attention guidance during AnimateDiff sampling (Liu et al., 22 Jul 2025).

This architecture suggests a general principle for motion transfer across heterogeneous objects: high-level semantics establish correspondence, low-level morphology enforces geometric compatibility, and temporal attention mediates the final generative control. In the reported formulation, the combination of semantic and morphological alignment is not an auxiliary refinement but the condition that makes attention-based motion guidance usable across large cross-object gaps (Liu et al., 22 Jul 2025).