Adaptive Caching for Faster Video Generation with Diffusion Transformers (2411.02397v2)

Abstract: Generating temporally-consistent high-fidelity videos can be computationally expensive, especially over longer temporal spans. More-recent Diffusion Transformers (DiTs) -- despite making significant headway in this context -- have only heightened such challenges as they rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. In this paper, we introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache), which is motivated by the fact that "not all videos are created equal": meaning, some videos require fewer denoising steps to attain a reasonable quality than others. Building on this, we not only cache computations through the diffusion process, but also devise a caching schedule tailored to each video generation, maximizing the quality-latency trade-off. We further introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, essentially controlling the compute allocation based on motion content. Altogether, our plug-and-play contributions grant significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video generation) without sacrificing the generation quality, across multiple video DiT baselines.

• The paper introduces Adaptive Caching, a training-free method that caches computations in Diffusion Transformers to reduce latency in video generation.
• It integrates Motion Regularization to dynamically allocate computing resources based on video motion, achieving up to a 4.7× speedup while maintaining quality.
• The plug-and-play implementation allows Easy integration into existing video synthesis systems without retraining, enhancing practical deployment.

Overview of Adaptive Caching for Faster Video Generation with Diffusion Transformers

This paper presents an innovative approach to accelerate video generation using Diffusion Transformers (DiTs) through a training-free method called Adaptive Caching (AdaCache). The authors address the computational challenges posed by high-fidelity, temporally-consistent video generation, especially over extended temporal spans. DiTs, although capable of producing high-quality results, challenge computational resources due to their large model sizes and attention mechanisms. AdaCache is designed to optimize the quality-latency trade-off by leveraging the inherent differences in video content, recognizing that some videos require fewer diffuse steps than others for acceptable quality.

Core Contributions

1. Adaptive Caching (AdaCache): This method involves caching and reusing computations in the diffusion process. AdaCache introduces a novel caching schedule that is tailored to each video generation, enabling it to achieve efficient performance without compromising quality.
2. Motion Regularization (MoReg): Integrated into AdaCache, MoReg utilizes video-specific information to allocate computing resources based on the motion content. This approach ensures that high-motion sequences receive more diffuse steps, optimizing computational efficiency and maintaining quality.
3. Plug-and-Play Implementation: The technique is characterized by its flexibility, allowing it to be integrated into existing video DiTs at inference time without requiring any retraining. This makes AdaCache a practical tool for speeding up video synthesis while maintaining or even enhancing generation quality.

Strong Numerical Results and Implications

The paper reports significant speedups in inference times, with AdaCache achieving up to a 4.7× acceleration in generating 720p videos while preserving quality. This is particularly evident in the context of high-resolution, long-form video generation, where the computational demands are most pronounced. These results were validated across multiple video DiT baselines, demonstrating that AdaCache consistently outperforms existing inference acceleration methods, such as PAB.

The results showcase how AdaCache strikes an effective balance between latency reduction and quality maintenance, evidenced by favorable results in both reference-free metrics like VBench and reference-based metrics such as PSNR, SSIM, and LPIPS.

Future Directions and Impact

AdaCache represents a significant step toward practical deployment of video DiTs by effectively reducing computational barriers. Its implications extend beyond video generation, potentially influencing a variety of applications in generative AI, where speedy and reliable processing is crucial. Moreover, its adaptability makes it suitable for diverse use-cases, including dynamic video editing and personalization tasks.

This work invites further exploration into sophisticated caching and regularization techniques that leverage content adaptiveness and motion dynamics. The potential for broader application of AdaCache across different generative models could spur advancements in real-time AI systems capable of handling complex visual tasks more efficiently.

In conclusion, AdaCache offers a compelling solution that optimally merges efficiency with performance, setting a benchmark for future research in video generation and beyond in the AI field.