Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation (2310.10769v1)

Published 16 Oct 2023 in cs.CV

Abstract: With the impressive progress in diffusion-based text-to-image generation, extending such powerful generative ability to text-to-video raises enormous attention. Existing methods either require large-scale text-video pairs and a large number of training resources or learn motions that are precisely aligned with template videos. It is non-trivial to balance a trade-off between the degree of generation freedom and the resource costs for video generation. In our study, we present a few-shot-based tuning framework, LAMP, which enables text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos on a single GPU. Specifically, we design a first-frame-conditioned pipeline that uses an off-the-shelf text-to-image model for content generation so that our tuned video diffusion model mainly focuses on motion learning. The well-developed text-to-image techniques can provide visually pleasing and diverse content as generation conditions, which highly improves video quality and generation freedom. To capture the features of temporal dimension, we expand the pretrained 2D convolution layers of the T2I model to our novel temporal-spatial motion learning layers and modify the attention blocks to the temporal level. Additionally, we develop an effective inference trick, shared-noise sampling, which can improve the stability of videos with computational costs. Our method can also be flexibly applied to other tasks, e.g. real-world image animation and video editing. Extensive experiments demonstrate that LAMP can effectively learn the motion pattern on limited data and generate high-quality videos. The code and models are available at https://rq-wu.github.io/projects/LAMP.

PDF HTML Abstract

Overview of "LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation"

The paper "LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation" presents an innovative approach for text-to-video (T2V) generation that focuses on mitigating the extensive data and computational resource requirements typically associated with this domain. Traditional T2V methodologies often necessitate large-scale datasets or rely heavily on template videos, thus limiting generative freedom and accessibility to researchers with limited computational resources. Addressing these challenges, the authors introduce a framework aptly named LAMP, which is designed to learn motion patterns from a compact set of videos using a minimal hardware configuration, specifically a single GPU.

Methodological Contributions

  1. First-Frame-Conditioned Pipeline: Central to LAMP's design is the first-frame-conditioned pipeline. This approach divides the T2V task into first-frame generation via a robust text-to-image (T2I) model and subsequent-frame prediction through a tuned video diffusion model. Notably, the use of pre-trained T2I models, such as Stable Diffusion XL (SD-XL), for generating the initial frame leverages their capacity for producing detailed and visually appealing content, which enhances overall video quality and generative freedom.
  2. Temporal-Spatial Motion Learning Layers: To efficiently capture temporal features essential for video coherence, the authors reconfigure the pre-trained 2D convolution layers of T2I models into temporal-spatial motion learning layers. These modifications allow simultaneous processing of spatial and temporal dimensions, utilizing a video-prediction-based 1D convolution strategy to maintain the integrity and thematic consistency of the generated motion across video frames.
  3. Shared-Noise Sampling: The paper also introduces a shared-noise sampling strategy, which further stabilizes the frame generation process by reducing noise variance across the video sequence. This technique facilitates enhanced consistency and quality in the generated videos without additional computational overhead.

Experimental Results

Through a series of exhaustive experiments, LAMP demonstrates remarkable proficiency in learning and generalizing motion patterns from a minimal dataset. Trained with only 8 to 16 videos, LAMP successfully generates high-quality videos that adhere closely to specified motion prompts while maintaining semantic relevance with new styles and objects. The results clearly indicate superior performance in terms of prompt alignment, consistency, and diversity, when compared to several state-of-the-art T2V generation techniques, including large-scale pre-trained models like AnimateDiff and methods like Tune-A-Video and Text2Video-Zero.

Implications and Future Directions

The implications of LAMP's framework are substantial for the field of video generation. By optimizing the use of limited data and computational resources, the proposed method democratizes access to high-quality video synthesis technologies. This advancement could indirectly facilitate wider adoption and exploration of generative models beyond well-funded research facilities.

Future research could explore extending LAMP's capabilities to handle more complex motion patterns and refine its learning layers for enhanced foreground-background separation. Additionally, distinct adjustments might be investigated to prevent overfitting in busy visual contexts, further expanding the versatility and robustness of few-shot-based video generation models.

Overall, LAMP represents a noteworthy contribution to the domain of text-to-video generation, offering a simpler yet potent method for learning and synthesizing motion patterns in a computationally constrained environment.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  2. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  3. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
  4. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  5. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  6. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023.
  7. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  8. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 34:23968–23982, 2021.
  9. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
  10. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  11. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  12. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  13. Large language models are frame-level directors for zero-shot text-to-video generation. arXiv preprint arXiv:2305.14330, 2023.
  14. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  15. A dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6121–6131, 2023.
  16. Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator. arXiv preprint arXiv:2309.14494, 2023.
  17. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017.
  18. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439, 2023.
  19. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  20. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104, 2016.
  21. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329, 2023.
  22. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  23. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  24. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  25. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535, 2023.
  26. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  27. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  28. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  29. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  30. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  31. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
  32. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
  33. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  34. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  35. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  36. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018, 2023.
  37. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623–7633, 2023.
  38. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036–5045, 2022.
  39. Rerender a video: Zero-shot text-guided video-to-video translation. arXiv preprint arXiv:2306.07954, 2023.
  40. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  41. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  42. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Ruiqi Wu (17 papers)
  2. Liangyu Chen (50 papers)
  3. Tong Yang (153 papers)
  4. Chunle Guo (30 papers)
  5. Chongyi Li (88 papers)
  6. Xiangyu Zhang (328 papers)
Citations (40)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. LAMP: Learn A Motion Pattern for Few-Shot Video Generation
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube