SmoothCache: Inference Acceleration for DiTs

• SmoothCache is a training-free caching method for Diffusion Transformers that reuses intermediate outputs across adjacent timesteps to speed up inference.
• It calibrates layer-wise representation errors offline using a thresholded reuse condition based on relative L1 error, enabling a fixed, model-specific caching schedule.
• Empirical results demonstrate speed-ups of up to 71% across various modalities, balancing reduced latency with maintained generation quality.

Searching arXiv for SmoothCache and directly related follow-up work. SmoothCache is a training-free caching method for Diffusion Transformers (DiTs) that speeds up denoising by reusing intermediate layer outputs across nearby diffusion timesteps. It is presented as a model-agnostic inference acceleration technique for DiT architectures, with the core premise that attention and feed-forward outputs often exhibit high similarity across adjacent steps of the diffusion trajectory. The method calibrates this smoothness on a small sample set, converts the measured representation errors into a fixed caching schedule, and then reuses cached sublayer outputs during inference without retraining or architectural modification. In the literature surveyed here, SmoothCache is defined primarily by the DiT-focused method of “SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers” (Liu et al., 2024), with a subsequent adaptation to F5-TTS demonstrating its transfer to diffusion transformer-based text-to-speech (Sakpiboonchit, 10 Sep 2025).

1. Conceptual basis and scope

SmoothCache addresses the cost structure of DiT inference. In diffusion inference, the denoiser is evaluated repeatedly over many timesteps, and each denoising step contains resource-intensive self-attention, cross-attention, and feed-forward modules. The method is built on the empirical observation that layer outputs at adjacent diffusion timesteps are very similar. The paper states this informally as $L_{t-k} \sim L_t$, then operationalizes the similarity with a relative $L_1$ error over calibration outputs (Liu et al., 2024).

The scope claimed for SmoothCache is “universal” in a specific sense. It is applied with the same calibration-and-threshold procedure to DiT-XL for image generation, Open-Sora for text-to-video, and Stable Audio Open for text-to-audio. This universality does not imply identical schedules, identical cache distances, or identical gains across models. Rather, it denotes a common mechanism: measure cross-timestep smoothness, define a thresholded reuse condition, and reuse cached outputs of expensive modules that immediately precede residual connections.

A common misconception is to treat SmoothCache as an online adaptive controller. The main paper does not do that. Its schedule is adaptive at calibration time, but fixed at runtime for a chosen model, solver, number of sampling steps, and threshold $\alpha$. A second misconception is to treat it as a training method. The method is explicitly training-free and does not require retraining, policy learning, or architecture-specific finetuning (Liu et al., 2024).

2. Formal method and cache decision rule

The formal core of SmoothCache is a calibrated estimate of layer-type-wise representation drift across timesteps. For layer type $i$, layer index $j$, current timestep $t$, and cached timestep $t+k$, the paper gives the operative criterion as

$$\mathcal{L}(L_{i_j,t}, L_{i_j,t+k}) \sim \frac{1}{N}\sum_{j=1}^{N}\frac{\| \tilde{L}_{i_j,t} - \tilde{L}_{i_j,t+k} \|_1}{\| \tilde{L}_{i_j,t} \|_1} < \alpha$$

where $N$ is the number of layers of type $i$, $L_1$0 denotes calibration outputs, and $L_1$1 is a single global threshold (Liu et al., 2024).

This formulation encodes several design decisions. First, the metric is a relative $L_1$2 error, normalized by the magnitude of the calibration activation. Second, the decision is made at the level of a layer type rather than independently for each layer instance. The paper motivates this grouping by noting that independently caching earlier layers perturbs the inputs of later layers, so a per-layer schedule derived under full recomputation may become inaccurate once caching is activated. Third, the runtime rule is threshold-based rather than learned. The paper initially introduces a hypothetical layer-dependent tolerance $L_1$3, but then collapses the method to a single threshold $L_1$4 to avoid costly per-layer hyperparameter search.

At inference, the system consults the precomputed schedule. If the calibrated error for a layer type at a timestep is below $L_1$5, the cached output from a nearby timestep is reused; otherwise the layer is recomputed and the cache is refreshed. The method targets outputs of modules immediately before residual additions, because this is the structural location at which replacing a submodule computation with a cached tensor is most natural in a residual block (Liu et al., 2024).

3. Calibration procedure and schedule construction

Calibration is central to SmoothCache. It is an offline or pre-inference procedure that estimates which timesteps and layer types are safe to cache for a particular inference configuration. The main paper uses 10 calibration samples for all evaluated modalities and reports that this sample count “does not matter much” for regenerating the same schedule at a fixed $L_1$6 (Liu et al., 2024).

For DiT-XL, calibration uses unconditional samples with the null prompt. For Open-Sora, calibration uses conditional 480p, 2s, 9:16 videos with prompts sampled from VidProM. For Stable Audio Open, calibration uses prompts sampled from the AudioCaps validation set. The calibration is solver-specific and step-count-specific. The paper evaluates DDIM for DiT-XL, Rectified Flow for Open-Sora, and DPM-Solver++(3M) SDE for Stable Audio Open, and the calibration schedule is intended to match those exact settings.

The allowable cache distances are small. For DiT-XL and Stable Audio Open, the paper uses $L_1$7, noting that beyond 4 timesteps the error grows too large. For Open-Sora, the method uses up to $L_1$8, particularly because some cross-attention components remain smooth at larger offsets. The resulting schedule is non-uniform across timesteps. The paper explicitly contrasts this with simple periodic skip schedules, arguing that cacheability is timestep-dependent and architecture-dependent.

A further implication is methodological rather than explicitly stated: calibration acts as a compressed summary of a model’s denoising dynamics. The paper reports narrow 95% confidence intervals for representation-error plots in the easier settings, while also noting that greater variance across calibration samples is associated with a narrower speed-quality Pareto frontier. This suggests that the reliability of a fixed reuse schedule depends materially on how stable those cross-timestep errors are across samples (Liu et al., 2024).

4. Architecture-specific instantiations

The concrete cache targets differ by architecture.

Model	Cached module types	Calibration notes
DiT-XL	Self-attention; Feed-forward	10 null-prompt samples; DDIM; $L_1$9
Open-Sora	Spatial and temporal self-attention, cross-attention, and FFN	10 VidProM prompts; Rectified Flow; up to $\alpha$0
Stable Audio Open	Self-attention; Cross-attention; Feed-forward	10 AudioCaps prompts; DPM-Solver++(3M) SDE; $\alpha$1

For DiT-XL, the paper reports that some components become more cacheable later in denoising. For Open-Sora, early and late timesteps are more sensitive, and the model is the most variance-sensitive of the three evaluated cases. For Stable Audio Open, the activation curves are smoother, which supports more aggressive reuse while maintaining quality. The paper also notes that the runtime decision structure remains static after calibration, which preserves compatibility with graph compilation and avoids the overhead of per-sample online similarity checks (Liu et al., 2024).

These architecture-specific differences are important for interpreting the term “model-agnostic.” SmoothCache is not architecture-blind; it is architecture-conditional through calibration. What is shared is the procedure, not the schedule.

5. Empirical performance and comparison landscape

The headline claim of the main paper is that SmoothCache achieves 8% to 71% speed up while maintaining or even improving generation quality across diverse modalities (Liu et al., 2024). The detailed experiments are more granular and reveal a clear speed-quality tradeoff controlled by $\alpha$2.

For DiT-XL-256x256 at 50 DDIM steps, the no-cache baseline reports FID $\alpha$3, sFID $\alpha$4, IS $\alpha$5, TMACs $\alpha$6, and latency $\alpha$7 s. With $\alpha$8, latency falls to $\alpha$9 s and TMACs to $i$0, while FID remains $i$1. With $i$2, latency drops to $i$3 s and TMACs to $i$4, with FID degrading to $i$5. With $i$6, latency reaches $i$7 s and TMACs $i$8, with FID $i$9. At 70 DDIM steps, the no-cache latency is $j$0 s and SmoothCache with $j$1 reduces it to $j$2 s, while FID moves from $j$3 to $j$4 (Liu et al., 2024).

For Open-Sora, gains are smaller. The baseline reports VBench $j$5, TMACs $j$6, and latency $j$7 s. SmoothCache with $j$8 gives VBench $j$9, TMACs $t$0, and latency $t$1 s; with $t$2, VBench is $t$3, TMACs $t$4, and latency $t$5 s. The paper attributes the narrower Pareto frontier to more complex spatiotemporal modeling, higher variance in representation errors, and larger non-DiT overheads (Liu et al., 2024).

For Stable Audio Open, the baseline reports TMACs $t$6 and latency $t$7 s. With $t$8, TMACs become $t$9 and latency $t+k$0 s. With $t+k$1, TMACs become $t+k$2 and latency $t+k$3 s, but quality degradation becomes more visible, especially on Song Describer. The paper also notes cases in which some quality metrics remain stable or slightly improve relative to baseline (Liu et al., 2024).

The comparison landscape is limited but informative. Against FORA, the paper states that SmoothCache generally gives similar or better quality at matched compute or latency points. Against L2C, L2C slightly outperforms on one DiT setting, but L2C requires training on ImageNet and retraining when the step count changes, whereas SmoothCache is training-free and portable.

6. Extensions, limitations, and later adaptations

The main limitations are explicit. SmoothCache relies on repeated DiT block structure and especially on residual connections after the bottleneck layers. It does not provide a formal theorem or a rigorous error-propagation bound. Calibration errors are measured under full recomputation, not under actual cached execution, so cascading approximation effects are only partially controlled by grouping decisions at the layer-type level. The paper also does not quantify memory overhead, although cached intermediate outputs necessarily consume memory. Gains depend on model compute profile: models with substantial non-DiT overhead, such as Open-Sora, exhibit smaller end-to-end acceleration (Liu et al., 2024).

A later adaptation to F5-TTS makes these dependency issues more concrete. “Accelerating Diffusion Transformer-Based Text-to-Speech with Transformer Layer Caching” integrates SmoothCache into F5-TTS, caches self-attention and feed-forward outputs before residual addition, and introduces a calibration phase using the $t+k$4 relative error between consecutive timesteps on 10 randomly sampled utterances from LibriSpeech-PC test-clean (Sakpiboonchit, 10 Sep 2025). That study reports that independently derived schedules for Attn and FFN produce artifacts when both are cached together, attributes this to inter-layer dependency through the residual pathway, and therefore adopts a unified Attn-based schedule for both layer types. It further imposes a rule of no more than 3 consecutive cached timesteps.

The F5-TTS results reinforce a central limitation already latent in the original SmoothCache formulation: safe reuse is not only layer-local but also path-dependent. At 32 NFE, the TTS adaptation reports RTF $t+k$5 with no cache, $t+k$6 at $t+k$7, and $t+k$8 at $t+k$9, while keeping WER, SIM-o, UTMOS, and NISQA close to baseline. At 16 NFE, moderate caching remains effective, but aggressive caching begins to degrade synthesis quality. The paper concludes that caching at higher denoising step counts reduces inference time without compromising output quality, whereas caching at lower steps can negatively impact synthesis quality similarly to reducing the total number of denoising steps (Sakpiboonchit, 10 Sep 2025).

Taken together, these works define SmoothCache as a calibration-driven, thresholded feature-reuse schedule for DiT inference. Its central abstraction is that diffusion-timestep smoothness can be converted into a static, solver-specific cache plan over attention and feed-forward computations. Its principal empirical significance lies in demonstrating that substantial portions of repeated DiT computation can be skipped by reusing nearby-timestep representations, while its principal open problem remains the control of inter-layer dependency and accumulated approximation error across architectures and sampling regimes.