MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generation

Abstract: Zero-shot Text-to-Video synthesis generates videos based on prompts without any videos. Without motion information from videos, motion priors implied in prompts are vital guidance. For example, the prompt "airplane landing on the runway" indicates motion priors that the "airplane" moves downwards while the "runway" stays static. Whereas the motion priors are not fully exploited in previous approaches, thus leading to two nontrivial issues: 1) the motion variation pattern remains unaltered and prompt-agnostic for disregarding motion priors; 2) the motion control of different objects is inaccurate and entangled without considering the independent motion priors of different objects. To tackle the two issues, we propose a prompt-adaptive and disentangled motion control strategy coined as MotionZero, which derives motion priors from prompts of different objects by Large-Language-Models and accordingly applies motion control of different objects to corresponding regions in disentanglement. Furthermore, to facilitate videos with varying degrees of motion amplitude, we propose a Motion-Aware Attention scheme which adjusts attention among frames by motion amplitude. Extensive experiments demonstrate that our strategy could correctly control motion of different objects and support versatile applications including zero-shot video edit.

• The paper introduces a framework using LLMs to extract object-level motion priors for improved zero-shot text-to-video synthesis.
• It employs disentangled motion control via region-specific warping, achieving superior semantic and temporal alignment.
• Experimental results show significant gains in textual alignment and motion correctness, validating its effectiveness in video generation.

MotionZero: Exploiting Motion Priors for Zero-shot Text-to-Video Generation

Introduction

The paper "MotionZero: Exploiting Motion Priors for Zero-shot Text-to-Video Generation" (2311.16635) addresses the intrinsic limitations of zero-shot text-to-video (T2V) generation pipelines that lack explicit modeling of object-centric motion priors implied by textual prompts. Previous approaches in zero-shot T2V either treat motion globally or rely on prompt decomposition into per-frame instructions, resulting in text-agnostic and often entangled motion trajectories, poor object control, and diminished semantic coherence across frames. MotionZero introduces a framework for prompt-adaptive and disentangled motion control, leveraging LLMs to extract object-level motion priors and constraining the generated video by these priors without the need for video training data.

Problem Formulation and Related Work

Zero-shot T2V operates on pretrained text-to-image (T2I) diffusion models (e.g., Stable Diffusion), repurposing them for video synthesis by controlling temporal progression across frames. Works such as Text2Video-Zero and DirecT2V integrate LLM-generated per-frame specifications or interpolate prompt sequences, but generally fail to accurately reflect complex, text-implied motion patterns. These strategies either induce a static, prompt-agnostic motion template (Text2Video-Zero) or encounter spatial and temporal incoherence due to lack of object-level disentanglement (DirecT2V, Free-Bloom).

MotionZero directly targets these gaps by using prompt and image understanding to extract explicit per-object motion priors and execute region-specific, disentangled warping in the latent space. This aligns video content more precisely with prompt semantics and allows multiple objects to be independently controlled.

Methodology

Extracting Motion Priors

MotionZero applies a two-stage process to extract and utilize motion priors:

1. LLM-based Motion Reasoning: Given a prompt, an LLM (e.g., GPT-4) is queried to decompose the scene into moving and static objects, as well as infer the canonical moving direction for each object. Explicit directionality is determined, partitioning object motion into discrete bins (e.g., up, down, left, right, and diagonals).
2. Disambiguating Non-directional Actions with Vision: For prompts with ambiguous verbs (e.g., “walking” or “riding”), the initial frame is synthesized using DDIM sampling from the T2I model. A Visual Question Answering (VQA) module infers likely movement direction from visual context, resolving ambiguities not explicit in text.

The resultant per-object motion plan is {Object: [Direction sequence], ...}, governing how each object should move per frame.

Disentangled Motion Control

Rather than warping the entire image globally, MotionZero leverages advanced open-vocabulary segmentation (SAM/Grounded-SAM) to obtain precise spatial masks for each controllable object in the initial frame. Motion control is then executed as follows:

• Spatial Warping: For each video frame, object-specific feature regions (as specified by masks) are warped according to the corresponding motion prior. Warp operations are reversed for the background to avoid producing visible artifacts and ensure proper occlusion/inpainting.
• Feature Fusion and Stochasticity: The disentangled, motion-controlled features are further processed via additional diffusion steps, ensuring high visual fidelity, stochastic variation across frames, and avoidance of hard “cut-and-paste” artifacts. The background is explicitly modeled as the “(c+1)-th character” with no motion apart from camera-induced or evolving context.

Motion-Aware Cross-frame Attention

Static cross-frame attention strategies, as in earlier zero-shot T2V works, are insufficient to handle diverse motion amplitudes. MotionZero introduces a Motion-Aware Attention (MAA) mechanism: The anchor frame for query-key-value attention is dynamically updated based on the IoU between current and previous object masks. Once the spatial configuration deviates significantly (tuned by a threshold $\gamma$), anchor frames are replaced—enabling high-mobility scenes to attend recent frames, and static scenes to anchor to the beginning for semantic consistency. This mechanism prevents issues such as object distortion and missing regions due to inappropriate temporal reference selection.

Applications

The granularity and modularity of MotionZero’s motion reasoning and control enable:

• Foreground/Background Semantic Editing: Independent semantic control enables zero-shot editing of either foreground or background while maintaining plausible interaction and visual harmony.
• Skeleton-based Body Control: LLM-inferred pseudo-skeleton trajectories, derived from text, can drive motion of articulated bodies (without explicit skeleton video supervision), exploiting ControlNet integration for fine-grained pose manipulation.
• Camera Motion Modeling: Relative motion between foreground and background can be used to mimic camera movements by applying inverse transforms to non-actor regions.
• Multi-stage/Evolving Event Video Synthesis: Complex prompts expressing event unfolding or scene transitions can be parsed into per-stage/slice motion plans, supporting temporally segmented video composition under LLM guidance.

Experimental Evaluation

Qualitative and Quantitative Results

MotionZero is compared to CogVideo, DirecT2V, and Text2Video-Zero across textual alignment (CLIP similarity), motion correctness (trajectory tracking versus ground-truth LLM-extracted motion), and user preference (human ratings on semantic consistency and visual quality). The framework demonstrates:

• Top CLIP Score: Achieves the highest frame-textual alignment, outperforming models trained with large-scale video data.
• Dominant Motion Correctness: Surpasses CogVideo, DirecT2V, and Zero by substantial margins in trajectory accuracy, e.g., $82.86\%$ (Ours) vs. $55.71\%$ (CogVideo).
• Clear Preference in Human Evaluation: MotionZero is favored by users for both textual alignment and video quality, especially for scenes with multiple, independently moving objects or articulated actions.

Ablation Studies

Removing LLM-extracted motion priors produces prompt-agnostic, erroneous trajectories; removing disentangled region control yields globally distorted or entangled motion; disabling MAA destabilizes object and background coherence, leading to object fusion or disappearance. Varying the IoU threshold for MAA reveals an optimal balance ($\gamma=0.6$), with too frequent anchor updates undermining appearance consistency and too infrequent updates causing drift.

Implications and Future Directions

By rigorously exploiting motion priors at the object and prompt level, MotionZero demonstrates a substantial leap in zero-shot video controllability and coherence without any additional video supervision or fine-tuning. This paradigm enables dynamic manipulation and editing of generated content, unifying visual grounding and linguistic reasoning. Future explorations should examine scaling to longer and more complex prompt sequences, incorporating granular 3D/trajectory-level motion priors, and extending cross-modal interaction (e.g., audio-driven/scene-aware video generation). Deeper integration with universal segmentation and grounding frameworks, as well as more advanced VQA pipelines, may further reduce errors in ambiguous settings or heavily cluttered scenes.

Conclusion

MotionZero provides a prompt-adaptive, disentangled framework for zero-shot T2V generation, robustly extracting and exploiting motion priors via LLMs and vision models. The method consistently outperforms prior works in prompt-aligned motion realization, multi-entity disentanglement, and editing flexibility. MotionZero sets a new reference for semantic-controlled, training-free T2V synthesis and opens new pathways for scalable, semantically compositional video generation in alignment with open-world textual descriptions (2311.16635).