Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation (2402.04324v2)

Published 6 Feb 2024 in cs.CV

Abstract: Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477, 2023.
  2. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1728–1738, 2021.
  3. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22563–22575, 2023.
  4. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769–31781, 2022.
  5. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9650–9660, 2021.
  6. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  23206–23217, 2023.
  7. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023a.
  8. Pixart-α𝛼\alphaitalic_α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023b.
  9. Seine: Short-to-long video diffusion model for generative transition and prediction. arXiv preprint arXiv:2310.20700, 2023c.
  10. Fine-grained open domain image animation with motion guidance. arXiv preprint arXiv:2311.12886, 2023.
  11. Stylevideogan: A temporal generative model using a pretrained stylegan. arXiv preprint arXiv:2107.07224, 2021.
  12. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp.  102–118. Springer, 2022.
  13. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  22930–22941, 2023.
  14. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023.
  15. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023.
  16. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
  17. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022.
  18. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  19. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  20. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  21. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
  22. Video diffusion models. arXiv:2204.03458, 2022b.
  23. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022.
  24. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657, 2023a.
  25. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982, 2023b.
  26. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439, 2023.
  27. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  28. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9801–9810, 2023.
  29. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  7083–7093, 2019.
  30. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  5404–5411, 2024.
  31. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329, 2023.
  32. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  33. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169, 2023.
  34. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  35. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  36. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10684–10695, 2022.
  37. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp.  234–241. Springer, 2015.
  38. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  39. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(10-11):2586–2606, 2020.
  40. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  41. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
  42. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3626–3636, 2022.
  43. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.  2256–2265. PMLR, 2015.
  44. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  45. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
  46. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  47. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp.  402–419. Springer, 2020.
  48. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104.15069, 2021.
  49. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp.  4489–4497, 2015.
  50. FVD: A new metric for video generation, 2019. URL https://openreview.net/forum?id=rylgEULtdN.
  51. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023a.
  52. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874, 2023b.
  53. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018, 2023c.
  54. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023d.
  55. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023e.
  56. Fairy: Fast parallelized instruction-guided video-to-video synthesis. arXiv preprint arXiv:2312.13834, 2023a.
  57. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021.
  58. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280, 2022.
  59. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  7623–7633, 2023b.
  60. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537, 2023c.
  61. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023.
  62. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5288–5296, 2016.
  63. Make pixels dance: High-dynamic video generation. arXiv preprint arXiv:2311.10982, 2023.
  64. Moonshot: Towards controllable video generation and editing with multimodal conditions. arXiv preprint arXiv:2401.01827, 2024.
  65. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023a.
  66. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023b.
  67. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Weiming Ren (12 papers)
  2. Ge Zhang (170 papers)
  3. Cong Wei (16 papers)
  4. Xinrun Du (23 papers)
  5. Wenhu Chen (134 papers)
  6. Huan Yang (306 papers)
  7. Wenhao Huang (98 papers)
Citations (39)
View on Semantic Scholar

Summary

  • The paper introduces a diffusion-based framework with spatiotemporal conditioning that enhances visual consistency in image-to-video generation.
  • It employs innovative noise initialization using low-frequency information to maintain consistent video layouts.
  • Comprehensive evaluations on the I2V-Bench benchmark demonstrate superior performance in visual quality and consistency over existing methods.

Introduction

The field of generative AI has seen impressive advancements in text-to-video (T2V) generation. Yet, existing methodologies exhibit limitations in achieving precise control over video content, a crucial aspect for practical applications. The novel work, titled "ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation," confronts this challenge head-on. The framework it introduces is a diffusion-based model that employs innovative spatiotemporal conditioning mechanisms to bolster visual consistency in video generation from a single given image and textual prompt.

Methodology

ConsistI2V brings to the table two main architectural innovations: the spatiotemporal attention applied over the initial frame and the distinctive method of noise initialization, which captures the low-frequency band of the initial frame to maintain the video's layout consistency. The model intricately weaves spatial and motion consistency throughout video generation by integrating cross-frame attention in spatial layers and local window attention operations in temporal layers. The devised framework, FrameInit, strategically uses low-frequency information during inference for noise initialization, substantially improving video stability and quality.

Evaluation

The authors introduce I2V-Bench, a comprehensive evaluation benchmark that allows for meticulous evaluation of I2V models against a wide array of metrics, capturing aspects such as visual quality and consistency. ConsistI2V's performance is rigorously assessed, both automatically and through human evaluation, across multiple datasets, including I2V-Bench. By dominating over existing methods in a majority of metrics and demonstrating outstanding visual consistency, ConsistI2V establishes itself as a significant contribution to controllable video generation.

Conclusion and Broader Impact

ConsistI2V marks an evolutionary stride in controlled video synthesis, adeptly addressing the need for visual consistency in I2V generation. Future work aims to improve further upon the model's efficacy by exploring more advanced training paradigms and high-quality datasets. Its broader impact lies in its potential for applications that demand high coherence and visual fidelity, such as virtual reality, filmmaking, and animated content creation, although the slower motion in some generated videos signals a need for further optimizations before broad deployment. With this paper, the authors set a high bar for future explorations in the domain of AI-driven video generation.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown