Training-free Diffusion Acceleration with Bottleneck Sampling (2503.18940v2)

Abstract: Diffusion models have demonstrated remarkable capabilities in visual content generation but remain challenging to deploy due to their high computational cost during inference. This computational burden primarily arises from the quadratic complexity of self-attention with respect to image or video resolution. While existing acceleration methods often compromise output quality or necessitate costly retraining, we observe that most diffusion models are pre-trained at lower resolutions, presenting an opportunity to exploit these low-resolution priors for more efficient inference without degrading performance. In this work, we introduce Bottleneck Sampling, a training-free framework that leverages low-resolution priors to reduce computational overhead while preserving output fidelity. Bottleneck Sampling follows a high-low-high denoising workflow: it performs high-resolution denoising in the initial and final stages while operating at lower resolutions in intermediate steps. To mitigate aliasing and blurring artifacts, we further refine the resolution transition points and adaptively shift the denoising timesteps at each stage. We evaluate Bottleneck Sampling on both image and video generation tasks, where extensive experiments demonstrate that it accelerates inference by up to 3$\times$ for image generation and 2.5$\times$ for video generation, all while maintaining output quality comparable to the standard full-resolution sampling process across multiple evaluation metrics.

• The paper introduces Bottleneck Sampling, a novel training-free framework to accelerate diffusion model inference, particularly at high resolutions, by exploiting low-resolution priors.
• Bottleneck Sampling utilizes a high-low-high denoising workflow with refined resolution transitions and an adaptive scheduler shift to achieve up to 3x speedup for image generation and 2.5x for video while maintaining quality.
• This training-free approach makes high-performance generative models more accessible, potentially mitigating deployment constraints for applications like video games and large-scale media production.

Training-free Diffusion Acceleration with Bottleneck Sampling

The paper under consideration introduces "Bottleneck Sampling," a novel framework designed to accelerate diffusion models used for visual content generation without requiring retraining. The primary focus is to address the inference inefficiency stemming from the quadratic complexity of self-attention mechanisms, which becomes prohibitive at high resolutions—a common trait in state-of-the-art diffusion models like Diffusion Transformers (DiTs).

Key Contributions

Bottleneck Sampling stands out through its training-free characteristic, capitalizing on the low-resolution priors inherent in diffusion models. The authors propose a unique high-low-high denoising workflow, which executes initial and final inference steps at high resolutions to preserve details and uses low-resolution computations in the intermediate steps to improve efficiency. This strategy is reinforced by two innovative techniques:

1. Resolution Transition Points: The framework refines the resolution change points to mitigate aliasing and blurring artifacts. By strategically injecting noise during these transitions, the process remains aligned with the spatial characteristics and maintains a consistent perceptual quality.
2. Adaptive Scheduler Shifting: Bottleneck Sampling includes an adaptive shift in the denoising scheduler during stage transitions. This addresses variations in the signal-to-noise ratio (SNR) across different resolutions, ensuring smoother and more stable denoising by focusing efforts on low-SNR regions.

Experimental Validation

The paper rigorously evaluates Bottleneck Sampling using two prominent diffusion transformer models: FLUX.1-dev for text-to-image generation and HunyuanVideo for text-to-video generation. The results are compelling, showing significant performance acceleration—up to threefold for image generation and 2.5 times for video generation—while maintaining output quality akin to the standard full-resolution approach. The framework efficiently bridges the gap between computational demands and performance, which is particularly valuable in resource-constrained settings.

Among several evaluation metrics, Bottleneck Sampling demonstrates superior capability in preserving text coherence and visual fidelity. It achieves high marks across various benchmarks like CLIP Score, ImageReward, and T2I-CompBench, indicating its robustness in dealing with challenging generative tasks, including intricate text rendering and complex compositional prompts.

Broader Implications and Future Work

Theoretically, the methodology intriguingly exploits the low-resolution pretrained priors without additional training, challenging the convention that high fidelity can only be achieved through extensive computational resources. Practically, this framework potentially mitigates the deployment constraints of high-performance generative models, making them more accessible for real-world applications such as video game graphics, virtual reality, and large-scale media content production.

Bottleneck Sampling's training-free nature poses intriguing possibilities for future developments in AI, particularly in optimizing inference strategies for machine learning models. Further refinements and explorations could involve investigating multi-stage configurations, adaptive upsampling techniques, or extending these principles to other modalities beyond image and video. The adaptability and efficiency showcased by Bottleneck Sampling might also inspire similar methodologies in related fields, such as natural language processing or audio synthesis, where model scalability and computational efficiency remain pivotal challenges.

In conclusion, Bottleneck Sampling offers a promising direction in the quest to balance efficiency and quality in the field of generative models, potentially paving the way for more sustainable and cost-effective deployment of AI technologies.