Papers
Topics
Authors
Recent
Search
2000 character limit reached

Pix2Gif: Motion-Guided Diffusion for GIF Generation

Published 7 Mar 2024 in cs.CV and cs.AI | (2403.04634v2)

Abstract: We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video) generation. We tackle this problem differently by formulating the task as an image translation problem steered by text and motion magnitude prompts, as shown in teaser fig. To ensure that the model adheres to motion guidance, we propose a new motion-guided warping module to spatially transform the features of the source image conditioned on the two types of prompts. Furthermore, we introduce a perceptual loss to ensure the transformed feature map remains within the same space as the target image, ensuring content consistency and coherence. In preparation for the model training, we meticulously curated data by extracting coherent image frames from the TGIF video-caption dataset, which provides rich information about the temporal changes of subjects. After pretraining, we apply our model in a zero-shot manner to a number of video datasets. Extensive qualitative and quantitative experiments demonstrate the effectiveness of our model -- it not only captures the semantic prompt from text but also the spatial ones from motion guidance. We train all our models using a single node of 16xV100 GPUs. Code, dataset and models are made public at: https://hiteshk03.github.io/Pix2Gif/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans, 2018.
  2. Blended latent diffusion. ACM Transactions on Graphics, 42(4):1–11, 2023.
  3. Future video prediction from a single frame for video anomaly detection, 2023.
  4. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023a.
  5. Align your latents: High-resolution video synthesis with latent diffusion models, 2023b.
  6. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  7. Maskgit: Masked generative image transformer, 2022.
  8. Muse: Text-to-image generation via masked generative transformers, 2023.
  9. Ilvr: Conditioning method for denoising diffusion probabilistic models, 2021.
  10. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:19822–19835, 2021.
  11. Taming transformers for high-resolution image synthesis, 2021.
  12. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  13. Guiding instruction-based image editing via multimodal large language models, 2023.
  14. Video representation learning through prediction for online object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 530–539, 2022.
  15. Preserve your own correlation: A noise prior for video diffusion models, 2023.
  16. Cater: A diagnostic dataset for compositional actions and temporal reasoning, 2020.
  17. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  18. Biomedjourney: Counterfactual biomedical image generation by instruction-learning from multimodal patient journeys. arXiv preprint arXiv:2310.10765, 2023.
  19. Latent video diffusion models for high-fidelity long video generation, 2023.
  20. Prompt-to-prompt image editing with cross attention control, 2022.
  21. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  22. Denoising diffusion probabilistic models, 2020.
  23. Video diffusion models, 2022.
  24. Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022.
  25. Make it move: Controllable image-to-video generation with text descriptions, 2022.
  26. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134, 2023.
  27. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  28. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.
  29. Auto-encoding variational bayes, 2022.
  30. Mutual suppression network for video prediction using disentangled features, 2019.
  31. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47–59, 2022.
  32. Tgif: A new dataset and benchmark on animated gif description, 2016.
  33. Gligen: Open-set grounded text-to-image generation, 2023.
  34. Future frame prediction for anomaly detection – a new baseline, 2018.
  35. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022.
  36. Sdedit: Guided image synthesis and editing with stochastic differential equations, 2022.
  37. Expanding language-image pretrained models for general video recognition, 2022.
  38. Conditional image-to-video generation with latent flow diffusion models, 2023.
  39. Action-conditional video prediction using deep networks in atari games, 2015.
  40. Folded recurrent neural networks for future video prediction, 2018.
  41. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  42. High-resolution image synthesis with latent diffusion models, 2022.
  43. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
  44. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  45. Convolutional lstm network: A machine learning approach for precipitation nowcasting, 2015.
  46. Very deep convolutional networks for large-scale image recognition, 2015.
  47. Make-a-video: Text-to-video generation without text-video data, 2022.
  48. Denoising diffusion implicit models, 2022.
  49. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
  50. Unsupervised learning of video representations using lstms, 2016.
  51. Fvd: A new metric for video generation. 2019.
  52. Pixel recurrent neural networks, 2016a.
  53. Conditional image generation with pixelcnn decoders, 2016b.
  54. Neural discrete representation learning, 2018.
  55. Decomposing motion and content for natural video sequence prediction, 2018.
  56. Phenaki: Variable length video generation from open domain textual description, 2022.
  57. Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022.
  58. Dreamvideo: High-fidelity image-to-video generation with image retention and text guidance. arXiv preprint arXiv:2312.03018, 2023a.
  59. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023b.
  60. Scaling autoregressive video models, 2020.
  61. Godiva: Generating open-domain videos from natural descriptions, 2021a.
  62. Nüwa: Visual synthesis pre-training for neural visual world creation, 2021b.
  63. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023.
  64. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  65. Reco: Region-controlled text-to-image generation, 2022.
  66. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.
  67. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459–10469, 2023a.
  68. Scaling autoregressive multi-modal models: Pretraining and instruction tuning, 2023b.
  69. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023a.
  70. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023b.
  71. 3d u-net: Learning dense volumetric segmentation from sparse annotation, 2016.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 4 tweets with 2 likes about this paper.

Sign Up for Free