Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VideoStudio: Generating Consistent-Content and Multi-Scene Videos (2401.01256v2)

Published 2 Jan 2024 in cs.CV and cs.CL

Abstract: The recent innovations and breakthroughs in diffusion models have significantly expanded the possibilities of generating high-quality videos for the given prompts. Most existing works tackle the single-scene scenario with only one video event occurring in a single background. Extending to generate multi-scene videos nevertheless is not trivial and necessitates to nicely manage the logic in between while preserving the consistent visual appearance of key content across video scenes. In this paper, we propose a novel framework, namely VideoStudio, for consistent-content and multi-scene video generation. Technically, VideoStudio leverages LLMs (LLM) to convert the input prompt into comprehensive multi-scene script that benefits from the logical knowledge learnt by LLM. The script for each scene includes a prompt describing the event, the foreground/background entities, as well as camera movement. VideoStudio identifies the common entities throughout the script and asks LLM to detail each entity. The resultant entity description is then fed into a text-to-image model to generate a reference image for each entity. Finally, VideoStudio outputs a multi-scene video by generating each scene video via a diffusion process that takes the reference images, the descriptive prompt of the event and camera movement into account. The diffusion model incorporates the reference images as the condition and alignment to strengthen the content consistency of multi-scene videos. Extensive experiments demonstrate that VideoStudio outperforms the SOTA video generation models in terms of visual quality, content consistency, and user preference. Source code is available at \url{https://github.com/FuchenUSTC/VideoStudio}.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. In ICCV, 2021.
  2. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In CVPR, 2023.
  3. Quo Vadis, Action Recognition? A New Model and The Kinetics Dataset. In CVPR, 2017.
  4. Diffusion Models Beat GANs on Image Synthesis. In NeurIPS, 2021.
  5. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In ACL, 2022.
  6. Structure and Content-guided Video Synthesis with Diffusion Models. In ICCV, 2023.
  7. TokenFlow: Consistent Diffusion Features for Consistent Video Editing. arXiv preprint arXiv:2307.10373, 2023.
  8. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. arXiv preprint arXiv:2307.04725, 2023.
  9. Latent Video Diffusion Models for High-Fidelity Long Video Generation. arXiv preprint arXiv:2211.13221, 2022.
  10. Gans Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS, 2017.
  11. Classifier-Free Diffusion Guidance. arXiv preprint arXiv:2207.12598, 2022.
  12. Denoising Diffusion Probabilistic Models. In NeurIPS, 2020.
  13. Imagen Video: High Definition Video Generation with Diffusion Models. arXiv preprint arXiv:2210.02303, 2022.
  14. CogVideo: Large-Scale Pretraining for Text-to-Video Generation via Transformers. In ICLR, 2023.
  15. VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet. arXiv preprint arXiv:2307.14073, 2023.
  16. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators. In ICCV, 2023.
  17. DeepStory: Video Story QA by Deep Embedded Memory Networks. In IJCAI, 2017.
  18. Dense-Captioning Events in Videos. In ICCV, 2017.
  19. StoryGAN: A Sequential Conditional GAN for Story Visualization. In CVPR, 2019.
  20. Contextual Transformer Networks for Visual Recognition. IEEE Trans. on PAMI, 2022.
  21. NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis. In NeurIPS, 2022.
  22. VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning. arXiv preprint arXiv:2309.15091, 2023.
  23. Grounding DINO: Marrying Dino with Grounded Pre-Training for Open-Set Object Detection. arXiv preprint arXiv:2303.05499, 2023.
  24. Gaussian Temporal Awareness Networks for Action Localization. In CVPR, 2019.
  25. Stand-Alone Inter-Frame Attention in Video Models. In CVPR, 2022a.
  26. Dynamic Temporal Filtering in Video Models. In ECCV, 2022b.
  27. Bi-calibration Networks for Weakly-Supervised Video Representation Learning. IJCV, 2023.
  28. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In NeurIPS, 2022.
  29. DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models. arXiv preprint arXiv:2211.01095, 2023.
  30. VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation. In CVPR, 2023.
  31. T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv preprint arXiv:2302.08453, 2023.
  32. Improved Denoising Diffusion Probabilistic Models. In ICML, 2021.
  33. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In ICML, 2022.
  34. OpenAI. GPT-4 Technical Report, 2023.
  35. CoDeF: Content Deformation Fields for Temporally Consistent Video Processing. arXiv preprint arXiv:2308.07926, 2023.
  36. FateZero: Fusing Attentions for Zero-shot Text-based Video Editing. In ICCV, 2023.
  37. U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection. Pattern Recognition, 2020.
  38. Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. In ICCV, 2017.
  39. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125, 2022.
  40. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.
  41. Laion-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. In NeurIPS, 2022.
  42. Edit-A-Video: Single Video Editing with Object-Aware Consistency. arXiv preprint arXiv:2303.07945, 2023.
  43. Make-a-video: Text-to-Video Generation without Text-Video Data. arXiv preprint arXiv:2209.14792, 2022.
  44. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In ICML, 2015.
  45. Denoising Diffusion Implicit Models. In ICLR, 2021.
  46. Generative Modeling by Estimating Gradients of the Data Distribution. In NeurIPS, 2019.
  47. FVD: A New Metric for Video Generation. In ICLR Workshop, 2019.
  48. Phenaki: Variable Length Video Generation from Open Domain Textual Description. In ICLR, 2023.
  49. MCVD-Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. In NeurIPS, 2022.
  50. Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising. arXiv preprint arXiv:2305.18264, 2023a.
  51. ModelScope Text-to-Video Technical Report. arXiv preprint arXiv:2308.06571, 2023b.
  52. VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation. arXiv preprint arXiv:2305.10874, 2023c.
  53. VideoComposer: Compositional Video Synthesis with Motion Controllability. In NeurIPS, 2023d.
  54. GODIVA: Generating Open-Domain Videos from Natural Descriptions. arXiv preprint arXiv:2104.14806, 2021.
  55. Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation. In ICCV, 2023.
  56. MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. In CVPR, 2016.
  57. Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning. In ECCV, 2022.
  58. Dual Vision Transformer. IEEE Trans. on PAMI, 2023.
  59. Dragnuwa: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory. arXiv preprint arXiv:2308.08089, 2023a.
  60. NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation. arXiv preprint arXiv:2303.12346, 2023b.
  61. GLM-130B: An Open Bilingual Pre-Trained Model. arXiv preprint arXiv:2210.02414, 2022.
  62. Adding Conditional Control to Text-to-Image Diffusion Models. In ICCV, 2023.
  63. Magicvideo: Efficient Video Generation with Latent Diffusion Models. arXiv preprint arXiv:2211.11018, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Fuchen Long (13 papers)
  2. Zhaofan Qiu (37 papers)
  3. Ting Yao (127 papers)
  4. Tao Mei (209 papers)
Citations (15)
View on Semantic Scholar