Beyond Few-Step Inference: Accelerating Video Diffusion Transformer Model Serving with Inter-Request Caching Reuse

Published 6 Apr 2026 in cs.CV | (2604.04451v1)

Abstract: Video Diffusion Transformer (DiT) models are a dominant approach for high-quality video generation but suffer from high inference cost due to iterative denoising. Existing caching approaches primarily exploit similarity within the diffusion process of a single request to skip redundant denoising steps. In this paper, we introduce Chorus, a caching approach that leverages similarity across requests to accelerate video diffusion model serving. Chorus achieves up to 45\% speedup on industrial 4-step distilled models, where prior intra-request caching approaches are ineffective. Particularly, Chorus employs a three-stage caching strategy along the denoising process. Stage 1 performs full reuse of latent features from similar requests. Stage 2 exploits inter-request caching in specific latent regions during intermediate denoising steps. This stage is combined with Token-Guided Attention Amplification to improve semantic alignment between the generated video and the conditional prompts, thereby extending the applicability of full reuse to later denoising steps.

Abstract PDF Upgrade to Chat

Authors (8)

Summary

The paper introduces Chorus, a three-stage inter-request caching technique that accelerates 4-step distilled video diffusion models by up to 45% while preserving quality.
It leverages Token-Guided Attention Amplification (TGAA) and Selective Region Denoising (SRD) to dynamically address semantic misalignment and optimize computational load.
Empirical results demonstrate significant improvements in latency and fidelity over traditional intra-request caching methods for large-scale video generation.

Inter-Request Caching for Accelerated Video Diffusion Transformer Serving

Introduction

The serving cost of Video Diffusion Transformer (DiT) models remains a major bottleneck for practical deployment in large-scale video generation. Existing acceleration solutions—particularly caching-based methods—focus on intra-request redundancy, where intermediate denoising states are reused within a single request's iterative process. This approach, while effective for non-distilled models with many denoising steps, is fundamentally inadequate for industrially distilled models operating in few-step (e.g., 4-step) regimes, where most intra-request redundancy has already been removed. The research introduces the Chorus framework, a three-stage inter-request caching approach. Chorus achieves significant acceleration by identifying and leveraging semantic similarity across requests, providing up to 45% end-to-end speedup on 4-step distilled models while maintaining high-fidelity video generation.

The primary technical novelty lies in Token-Guided Attention Amplification (TGAA), which mitigates semantic misalignment introduced by inter-request caching. Furthermore, Selective Region Denoising (SRD) extends partial computation in intermediate steps, realizing a flexible trade-off between efficiency and quality.

Figure 2: Chorus overview, showing full reuse, selective region reuse, and full computation across three denoising stages.

Video Diffusion Transformers and Serving Bottlenecks

Video DiTs such as Wan2.1 and Hunyuan integrate denoising diffusion processes with Transformer architectures, processing a video as a long sequence and employing iterative refinement over dozens of timesteps. The high computational cost results from the quadratic scaling of self- and cross-attention, dominance of attention computations (up to 80% of runtime at 720p), and absence of effective batch processing.

Distillation compresses these models and enables reduction to 4–8 denoising steps with marginal quality degradation, representing the practical regime of industrial deployments. In this low-step setting, existing intra-request caching and pruning strategies lose efficacy, as confirmed by empirical benchmarks.

Figure 3: DiT model architecture and the iterative inference process.

Inter-Request Caching: From Images to Video

Inter-request caching exploits redundancy across requests sharing similar prompts or semantic content. Prior work such as NIRVANA demonstrates that image-generation services receive frequent semantically-similar prompt clusters, enabling retrieval and effective reuse of intermediate states. Extending this to video with DiTs introduces unique challenges: temporal structure drastically amplifies sensitivity to semantic misalignment, and feasible reuse is limited to the earliest denoising steps.

Figure 4: Intra- and inter-request caching reuse, contrasting redundancy within and across requests with explicit denoising timetable mapping.

Chorus extends this paradigm with a three-stage hierarchical policy:

Stage 1 — Full Reuse: Retrieve a cache entry with a prompt embedding (via CLIP similarity) matching the target, and initialize denoising from the stored intermediate latent.
Stage 2 — Selective Region Reuse (SRD): Identify and recompute only regions with semantic divergence, reusing matched regions from cache; realize computational sparsity via dynamic latent masks.
Stage 3 — Full Compute: Disable all reuse for final refinement steps, repairing discontinuities and local detail artifacts.

This procedure is modulated by CLIP-based semantic similarity, adaptively partitioning denoising stages for each request.

Figure 1: Example of Chorus video generation on a 4-step distilled Wan2.1. Cached latents from a semantically-similar request are reused; semantic correctness and fidelity are preserved.

Token-Guided Attention Amplification (TGAA)

The principal source of quality degradation in inter-request reuse is semantic misalignment: reused latents overfit to cached prompt attributes, failing to align with the target semantics due to the video model's strong temporal priors.

(TGAA)—a training-free inference-time mechanism—solves this by identifying differential tokens (via an LLM) between source/target prompts. It amplifies their cross-attention key vectors and the global cross-attention output during the denoising process. Amplification is dynamically decayed across steps, tied to both denoising progress and match strength:

Key Amplification: Scales cross-attention keys of differential tokens by a factor $\gamma_k$ , forcing the model to prioritize semantic differences.
Output Amplification: Scales cross-attention output by $\gamma_o$ , boosting textual control and semantic steering.
Figure 5: Key amplification in TGAA highlights condition tokens most responsible for semantic shifts between prompts, driving alignment in the latent denoising process.

Extensive ablation reveals that TGAA enables significantly deeper reuse into later denoising steps, and optimal amplification must be scheduled to prevent artifacts (motion degradation, overshooting, blurriness).

Figure 6: Cross-attention maps at early (step 2) and late (step 48) stages. Conditional influence is strongest early, supporting frontloaded amplification.

Figure 8: Sensitivity of TGAA amplification factors; incorrect scheduling leads to motion artifacts, prompt leakage, and failed steering.

Selective Region Denoising (SRD)

As denoising progresses and generation becomes highly conditional, full latent reuse is often too coarse, introducing cross-frame or object-level inconsistencies. SRD addresses this via localized computation:

Divergent regions are first detected by LLM-driven token comparison and grounding via open-vocabulary segmentation (e.g., GroundedSAM) on stored cache video frames.
Binary masks are generated, dilated to create hierarchical 'edit' and 'visible' regions. The latter provides a contextual window for Transformer computation, ensuring seamless fusion.
Only tokens inside these masks are recomputed during attention and MLP operations, with cache states reused elsewhere.
SRD is directly implemented within FlashAttention kernels to take advantage of block-sparse attention, translating sparsity to effective speedup.
Figure 9: Visual depiction of selective region denoising: only object-differentiating areas trigger fresh computation, with background and context regions reused from cache.

Figure 10: Mask generation pipeline — object extraction via LLM, segmentation on source video, projection to the latent grid for denoising.

SRD yields a tunable compute–quality trade-off (precision controlled by mask size), and in practical settings enables up to 1.6× speedup per denoising step with minimal quality reduction. Aggressive masking can produce artifacts at boundaries but moderate expansion preserves fidelity.

Experimental Results

Empirical evaluation on large-scale T2V (e.g., Wan2.1-14B and its 4-step distilled variant) using real-world prompt datasets reveals several strong claims:

Chorus outperforms prior inter-request baseline NIRVANA in both generation quality (CLIP-SCORE and VBench) and end-to-end speed. On 4-step models, Chorus is uniquely effective, with a peak measured speedup of 1.45×—where intra-request caching is entirely ineffective.
On 50-step vanilla models, Chorus is orthogonal to intra-request caching; combining it with TeaCache yields up to 1.94×/3.09× combined speedups for 14B/1.3B models, respectively. This shows additive benefits when both forms of redundancy are exploited.
Cache hit rates (i.e., match probability) scale with cache size and prompt locality, with higher acceleration as system state matures.
Figure 7: Cache hit rate and average latency improvement as the cache grows under cold start conditions.

Ablations confirm that TGAA (both key and output amplification) directly mitigates semantic drift introduced by naive latent reuse, and that SRD offers a controlled efficiency–accuracy trade-off.

Practical and Theoretical Implications

Chorus's ability to exploit inter-request similarity at industrial scales redefines video DiT serving efficiency limits. The approach:

Delivers practical acceleration in real low-step (distilled) settings where existing methods saturate.
Establishes that inter-request redundancy, unlike intra-request, remains exploitable and scalable with cache size and prompt distribution coverage.
Introduces a flexible hybrid of full/partial computation, setting a blueprint for future content- and region-adaptive inference in generative media.

Theoretically, the results indicate that semantic similarity in prompt space translates robustly into latent space alignment, provided input-adaptive amplification mechanisms such as TGAA are applied to manage temporal priors. This insight extends applicability to foundation video models and may inform the design of future generative systems where serving cost is a dominant concern.

Conclusion

Chorus is a significant advancement in accelerating video diffusion model inference. By unifying inter-request caching, adaptive token-guided attention modulation, and selective region denoising, it achieves substantial speedup on both distilled and non-distilled regimes of Video DiT serving, surpassing both pure intra-request and prior inter-request caching methods in quality-efficiency trade-off. As model and dataset scale increase, inter-request similarity will become even more valuable, and techniques like Chorus will be integral to deploying generative video models at production scale.