Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 158 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 36 tok/s Pro
GPT-5 High 35 tok/s Pro
GPT-4o 112 tok/s Pro
Kimi K2 177 tok/s Pro
GPT OSS 120B 452 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generation (2311.16635v1)

Published 28 Nov 2023 in cs.CV

Abstract: Zero-shot Text-to-Video synthesis generates videos based on prompts without any videos. Without motion information from videos, motion priors implied in prompts are vital guidance. For example, the prompt "airplane landing on the runway" indicates motion priors that the "airplane" moves downwards while the "runway" stays static. Whereas the motion priors are not fully exploited in previous approaches, thus leading to two nontrivial issues: 1) the motion variation pattern remains unaltered and prompt-agnostic for disregarding motion priors; 2) the motion control of different objects is inaccurate and entangled without considering the independent motion priors of different objects. To tackle the two issues, we propose a prompt-adaptive and disentangled motion control strategy coined as MotionZero, which derives motion priors from prompts of different objects by Large-Language-Models and accordingly applies motion control of different objects to corresponding regions in disentanglement. Furthermore, to facilitate videos with varying degrees of motion amplitude, we propose a Motion-Aware Attention scheme which adjusts attention among frames by motion amplitude. Extensive experiments demonstrate that our strategy could correctly control motion of different objects and support versatile applications including zero-shot video edit.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477, 2023.
  2. Conditional gan with discriminative filter generation for text-to-video synthesis. In IJCAI, 2019.
  3. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
  4. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
  5. Language models are few-shot learners. NeurIPS, 2020.
  6. Segment and track anything. arXiv preprint arXiv:2305.06558, 2023.
  7. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In NeurIPS, 2022.
  8. Diffsynth: Latent in-iteration deflickering for realistic video synthesis. arXiv preprint arXiv:2308.03463, 2023.
  9. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023.
  10. Empowering dynamics-aware text-to-video diffusion with large language models. arXiv preprint arXiv:2308.13812, 2023.
  11. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393, 2023.
  12. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, 2023.
  13. Animate-a-story: Storytelling with retrieval-augmented video generation. arXiv preprint arXiv:2307.06940, 2023.
  14. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022.
  15. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  16. Denoising diffusion probabilistic models. NeurIPS, 2020.
  17. Video diffusion models. In NeurIPS, 2022.
  18. Large language models are frame-level directors for zero-shot text-to-video generation. arXiv preprint arXiv:2305.14330, 2023.
  19. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2023.
  20. Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator. arXiv preprint arXiv:2309.14494, 2023.
  21. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022.
  22. Text2performer: Text-driven human video generation. arXiv preprint arXiv:2304.08483, 2023.
  23. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439, 2023.
  24. Segment anything. arXiv:2304.02643, 2023.
  25. Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation. arXiv preprint arXiv:2309.00398, 2023.
  26. Video generation from text. In AAAI, 2018.
  27. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023.
  28. Llm-grounded video diffusion models. arXiv preprint arXiv:2309.17444, 2023.
  29. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091, 2023.
  30. Dual-stream diffusion net for text-to-video generation. arXiv preprint arXiv:2308.08316, 2023.
  31. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  32. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761, 2023.
  33. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795, 2023.
  34. Follow your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01186, 2023.
  35. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329, 2023.
  36. OpenAI. Gpt-4 technical report, 2023.
  37. Training language models to follow instructions with human feedback. NeurIPS, 2022.
  38. To create what you tell: Generating videos from captions. In ACM MM, 2017.
  39. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05427, 2023.
  40. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535, 2023.
  41. Dancing avatar: Pose and text-guided human motion videos synthesis with image diffusion model. arXiv preprint arXiv:2308.07749, 2023.
  42. Learning transferable visual models from natural language supervision. In ICML, 2021.
  43. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  44. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  45. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In CVPR, 2023.
  46. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
  47. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023.
  48. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
  49. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  50. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
  51. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023.
  52. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023.
  53. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874, 2023.
  54. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018, 2023.
  55. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023.
  56. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021.
  57. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022.
  58. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023.
  59. Make-your-video: Customized video generation using textual and structural guidance. arXiv preprint arXiv:2306.00943, 2023.
  60. Simda: Simple diffusion adapter for efficient video generation. arXiv preprint arXiv:2308.09710, 2023.
  61. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872, 2023.
  62. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303.12346, 2023.
  63. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023.
  64. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
  65. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023.
  66. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2022.
  67. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
  68. Moviefactory: Automatic movie creation from text using large generative models for language and images. In ACM MM BNI, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube