DeepCache: Accelerating Diffusion Models for Free (2312.00858v2)

Abstract: Diffusion models have recently gained unprecedented attention in the field of image synthesis due to their remarkable generative capabilities. Notwithstanding their prowess, these models often incur substantial computational costs, primarily attributed to the sequential denoising process and cumbersome model size. Traditional methods for compressing diffusion models typically involve extensive retraining, presenting cost and feasibility challenges. In this paper, we introduce DeepCache, a novel training-free paradigm that accelerates diffusion models from the perspective of model architecture. DeepCache capitalizes on the inherent temporal redundancy observed in the sequential denoising steps of diffusion models, which caches and retrieves features across adjacent denoising stages, thereby curtailing redundant computations. Utilizing the property of the U-Net, we reuse the high-level features while updating the low-level features in a very cheap way. This innovative strategy, in turn, enables a speedup factor of 2.3$\times$ for Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1$\times$ for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments also demonstrate DeepCache's superiority over existing pruning and distillation methods that necessitate retraining and its compatibility with current sampling techniques. Furthermore, we find that under the same throughput, DeepCache effectively achieves comparable or even marginally improved results with DDIM or PLMS. The code is available at https://github.com/horseee/DeepCache

This paper introduces DeepCache, a training-free method designed to accelerate the inference speed of diffusion models like Stable Diffusion, Latent Diffusion Models (LDM), and Denoising Diffusion Probabilistic Models (DDPM). The core problem addressed is the significant computational cost associated with the sequential denoising process in these models. Unlike methods requiring retraining or fine-tuning (e.g., distillation, pruning), DeepCache modifies the inference process dynamically at runtime.

Core Observation and Idea:

The authors observe that during the iterative denoising process, the high-level features computed by the deeper layers of the U-Net architecture exhibit significant temporal similarity between adjacent timesteps. This means that computing these features repeatedly in consecutive steps involves redundant calculations. DeepCache leverages this redundancy by caching and reusing these high-level features.

Methodology:

DeepCache utilizes the inherent structure of the U-Net, specifically its encoder-decoder structure with skip connections.

1. Caching High-Level Features: At certain timesteps (cache update steps), the model performs a full forward pass through the U-Net. During this pass, the output features from an up-sampling block $U_{m+1}$ (which represent high-level, processed information from the deeper layers) are stored in a cache.

   $F^t_{\text{cache}} \leftarrow U_{m+1}^t(\cdot)$

2. Retrieving and Partial Inference: In the subsequent step(s) (retrieve steps), instead of running the full U-Net, DeepCache performs a partial inference:
   • It computes only the low-level features from the corresponding down-sampling block $D_m$ in the encoder path using the current noisy input $x_{t-1}$. This computation is relatively cheap as it involves only the shallower layers up to $D_m$.
   • It retrieves the cached high-level features $F^t_{\text{cache}}$ from the previous step $t$.
   • It concatenates the newly computed low-level features $D^{t-1}_m(\cdot)$ with the retrieved high-level features $F^t_{\text{cache}}$ and feeds them into the up-sampling block $U_m$.

   $$\operatorname{Concat}(D^{t-1}_m(\cdot), F_{\text{cache}}^t)$$

   The rest of the up-sampling path ($U_m$ down to $U_1$) is computed normally. This avoids recomputing the computationally expensive deeper parts of the U-Net ($U_{m+1}$ and deeper).

Implementation Strategies:

• 1:N Inference (Uniform): The simplest strategy involves performing one full inference step (cache update) followed by $N-1$ partial inference steps (retrieve steps) using the same cached features. The sequence of full inference steps is $$\mathcal{I} = \{ iN \mid 0 \leq i < \lceil T / N \rceil \}$$, where $T$ is the total number of denoising steps. Increasing $N$ increases the speedup but can potentially degrade quality.

• Non-uniform 1:N Inference: Acknowledging that feature similarity isn't constant across all timesteps (it often decreases significantly around certain points in the denoising process), this strategy performs full updates more frequently around timesteps where similarity is expected to be lower. The timesteps for full inference $\mathcal{I}$ are chosen using a power function centered around a timestep $c$:

  $$\mathcal{L} = \{l_i \mid l_i \in \operatorname{linear\_space}((-c)^{\frac{1}{p}}, (T - c)^{\frac{1}{p}}, k) \}$$

  $$\mathcal{I} = \operatorname{unique\_int}(\{i_k \mid i_k=(l_k)^p+c, \text { where } l_k \in \mathcal{L}\})$$

  Here, $p$ (power) and $c$ (center) are hyperparameters. This strategy aims to improve quality compared to uniform caching, especially for larger $N$.

Pseudocode Overview (Simplified):

def deepcache_step(x_t, t, cache, m): # --- Cache Update Step --- # Full U-Net pass h = [x_t] for i in range(1, d + 1): h.append(D[i](h[-1])) # Down path u_next = M(h[d]) # Middle for i in range(d, 0, -1): if i == m: cache['u_next'] = u_next # Cache feature before U_m u_i = U[i](concat(u_next, h[i])) u_next = u_i epsilon_pred = u_next # Final prediction x_prev = compute_x_prev(x_t, t, epsilon_pred) # Standard DDIM/PLMS update return x_prev, cache def deepcache_retrieve_step(x_t, t, cache, m): # --- Retrieve Step --- # Partial U-Net pass (only up to D_m) h = [x_t] for i in range(1, m + 1): h.append(D[i](h[-1])) u_next = cache['u_next'] # Retrieve cached feature # Compute remaining up path from U_m for i in range(m, 0, -1): u_i = U[i](concat(u_next, h[i])) u_next = u_i epsilon_pred = u_next # Final prediction x_prev = compute_x_prev(x_t, t, epsilon_pred) # Standard DDIM/PLMS update return x_prev cache = {} x = initial_noise for t in range(T, 0, -1): if (T - t) % N == 0: # Cache update step x, cache = deepcache_step(x, t, cache, m) else: # Retrieve step x = deepcache_retrieve_step(x, t, cache, m)

Experimental Results:

• DeepCache demonstrated significant speedups: 2.3x for Stable Diffusion v1.5 (50 PLMS steps) with minimal quality drop (0.05 CLIP Score), and up to 7.0x-10.5x for LDM-4-G (250 DDIM steps) with moderate quality drop (e.g., FID from 3.37 to 4.41 at 7.0x speedup using uniform N=10, or to 4.27 using non-uniform N=10).

• It outperformed retraining-based compression methods like Diff-Pruning and BK-SDM variants in terms of quality at comparable or higher throughputs.

• DeepCache is compatible with existing fast samplers like DDIM and PLMS. When compared to reducing sampler steps (e.g., using 25 PLMS steps vs. 50), DeepCache often achieved comparable or slightly better quality at similar throughputs.

• Ablation studies confirmed the importance of reusing cached features and the positive impact of the partial inference steps compared to simply skipping steps.

• The non-uniform strategy significantly improved results over the uniform one for larger caching intervals (e.g., N=10, N=20).

Practical Implementation Considerations:

• Training-Free: Easy to integrate into existing inference pipelines for pre-trained U-Net based diffusion models without any model retraining.

• Hyperparameters:

  • m (Skip Branch Index): Controls the trade-off between speedup and quality. Caching at shallower branches (smaller m) gives more speedup but potentially lower quality, as more of the network is skipped. Figure 3 shows MACs per branch, helping guide this choice.
  • N (Caching Interval): Controls the frequency of cache updates. Larger N means more speedup but potentially lower quality. Optimal N seems to be model/dataset dependent, often effective up to N=5 or N=10.
  • c, p (Non-uniform Strategy): Requires tuning for optimal performance if using the non-uniform strategy, especially for large N. Appendix B provides guidance.
• Computational Cost: Reduces MACs significantly by skipping deeper layers during retrieve steps. The actual speedup depends on the chosen branch m and the U-Net architecture's computational distribution (Figure 3).
• Memory: Requires storing the cached feature tensor ($F^t_{\text{cache}}$), which adds a memory overhead compared to standard inference.
• Limitations: Effectiveness depends on the U-Net structure; if shallow skip branches still contain a large portion of the computation, speedup is limited. Very large N values (e.g., N=20) can lead to noticeable quality degradation.

In summary, DeepCache offers a practical, training-free method to accelerate diffusion model inference by exploiting temporal redundancy in U-Net features. It provides a tunable trade-off between speed and quality, is compatible with existing samplers, and often outperforms retraining-based compression methods at similar throughputs.