Efficient Video Diffusion Models

• Efficient video diffusion models are advanced systems that lower high-dimensional video processing costs through optimized denoising and latent decomposition.
• They employ strategies such as step distillation, efficient attention, model compression, and caching to reduce computational, memory, and latency overheads.
• These methods balance quality–latency trade-offs and enable real-time, mobile, and distributed deployments by integrating pruning, quantization, and structured temporal modeling.

Searching arXiv for papers on efficient video diffusion models and related acceleration methods. Efficient video diffusion models are video-generation or video-restoration systems that retain the probabilistic formulation and iterative denoising behavior of diffusion models while explicitly reducing the computational, memory, latency, or communication costs that arise from high-dimensional spatiotemporal processing. In the literature summarized here, efficiency is pursued through a small number of recurrent strategies: reducing the number of denoising steps, reducing per-step token or latent complexity, replacing quadratic temporal modules, compressing weights or activations, reusing intermediate features, and restructuring inference or serving. A deployment-oriented survey organizes these directions into four main paradigms—step distillation, efficient attention, model compression, and cache/trajectory optimization—and frames them around two core objectives: reducing the number of function evaluations and minimizing per-step overhead (Shao et al., 17 Apr 2026).

1. Problem setting and efficiency bottlenecks

Video diffusion models inherit the iterative denoising procedure of image diffusion, but video synthesis compounds cost across temporal length, spatial resolution, and repeated function evaluations. One paper states that current video diffusion models are “still limited by the high memory and computational requirements” because they “often attempt to process high-dimensional videos directly” (Yu et al., 2024). Another notes that recent models for video generation have “predominantly utilized attention layers to extract temporal features,” but that these layers are limited by computational costs that “increase quadratically with the sequence length,” which becomes especially restrictive for long video generation (Oshima et al., 2024).

The survey on efficient video diffusion models identifies two recurring cost sources. First, iterative sampling requires many denoising evaluations; second, each evaluation is itself expensive because video generation operates over large spatiotemporal token sets, so attention and memory traffic become dominant bottlenecks (Shao et al., 17 Apr 2026). This framing appears across diverse settings. In text-to-video generation, “processing multiple frame features concurrently, coupled with the considerable model size, results in high latency and extensive memory consumption” (Tian et al., 2024). In mobile deployment, the same issue is described as the high computational demand that restricts video diffusion models’ use on mobile devices (Yahia et al., 2024). In multi-GPU serving, the problem appears as cubic scaling of memory consumption relative to 3D spatiotemporal attention and the resulting communication bottlenecks (Wu et al., 8 Dec 2025).

A common misconception is that efficiency is equivalent to reducing denoising steps alone. The surveyed work suggests a broader picture. Some methods reduce the number of steps to one, four, or five; others keep the diffusion horizon but lower per-step cost through latent-space factorization, temporal state-space modeling, feature reuse, block-wise autoregression, quantization, or network architecture search. A plausible implication is that efficient video diffusion is better understood as a systems-and-model co-design problem rather than a single algorithmic trick (Shao et al., 17 Apr 2026).

2. Latent decomposition and compact video representations

A major line of work reduces cost by replacing full spatiotemporal diffusion with compact latent representations that separate static and dynamic information. “Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition” introduces the content-motion latent diffusion model (CMD), which encodes a video as a single content frame and a low-dimensional motion latent representation (Yu et al., 2024). The content frame is defined by learned attention weights,

$$s = f_{\phi_I}(u)\in\mathbb R^{L\times 1\times 1\times 1}, \quad \alpha_\ell = \mathrm{softmax}_\ell(s)\,, \quad \bar x = \sum_{\ell=1}^L \alpha_\ell \,x^\ell,$$

while the motion code is represented by two small planes $z_x\in\mathbb R^{D\times L\times H'}$ and $z_y\in\mathbb R^{D\times L\times W'}$ (Yu et al., 2024). CMD generates the content frame by fine-tuning a pretrained image diffusion model and the motion latent by training a lightweight diffusion model; because it “never processes full $L$-frame tensors in the diffusion steps,” it yields major savings (Yu et al., 2024). The reported results state that CMD can sample “a video of $512\times1024$ resolution and length 16 in 3.1 seconds,” is “7.7$\times$ faster than prior approaches,” and needs “only 47 TFLOPs and 5.6 GB,” compared with “626 TFLOPs and 8.4 GB” for LVDM and “∼939 TFLOPs and 8.5 GB GPU” for ModelScope (Yu et al., 2024). On WebVid-10M, the same summary reports “FVD = 238.3 vs. the previous best 292.4,” while the abstract states “an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous state-of-the-art of 292.4” (Yu et al., 2024).

Projected latent factorization appears in a different form in “Video Probabilistic Diffusion Models in Projected Latent Space” (Yu et al., 2023). PVDM projects a video into three coupled 2D latent maps: a static plane $z^{\rm s}$, a height-time plane $z^{\rm h}$, and a width-time plane $z^{\rm w}$ (Yu et al., 2023). This reduces token complexity from a full 3D grid to $O(H'W' + T\,W' + T\,H')$ latent tokens. The summary reports that on a single 3090Ti at $z_x\in\mathbb R^{D\times L\times H'}$0, “PVDM-L (200/200–step)” performs 16-frame inference in “~20.4 s & 5.22 GB,” while “PVDM-S (100/20–step)” takes “~7.9 s & 4.33 GB,” compared with a pixel-space VDM that is “OOM at 24 GB” for training and requires “~113 s & 11.1 GB” for 16-frame inference (Yu et al., 2023). On UCF-101, the reported values are “FVD$z_x\in\mathbb R^{D\times L\times H'}$1: StyleGAN-V 1431.0 → PVDM-L 398.9 → PVDM-S 457.4” and “FVD$z_x\in\mathbb R^{D\times L\times H'}$2: StyleGAN-V 1773.4 → PVDM-L 639.7 → PVDM-S 902.2” (Yu et al., 2023).

Efficiency can also be shifted upstream into the video autoencoder. “LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models” replaces heavier video VAE blocks with a Residual Neighborhood-Aware Feedforward module and non-overlapping patch operations, augmented by Haar wavelets and compressed sensing (Cheng et al., 18 Mar 2025). The summary states that this design yields “up to 50× fewer FLOPs and 44× faster inference,” with a 17-frame $z_x\in\mathbb R^{D\times L\times H'}$3 encode/decode time of “0.46 s” versus “20.26 s” for VidTok on a single NVIDIA A40, and a memory footprint of “≈2.3 GB vs ≈13.8 GB” (Cheng et al., 18 Mar 2025). This suggests that latent efficiency is not limited to the denoiser; encoder-decoder design can materially determine end-to-end cost.

3. Temporal modeling beyond quadratic attention

Another central strategy is to replace or reorganize temporal computation so that long video generation scales more favorably with sequence length. “SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces” replaces temporal attention with bidirectional structured state-space model blocks (Oshima et al., 2024). The temporal block is defined by a forward and backward SSM stream,

$z_x\in\mathbb R^{D\times L\times H'}$4

followed by

$z_x\in\mathbb R^{D\times L\times H'}$5

The reported complexity comparison is explicit: self-attention has time $z_x\in\mathbb R^{D\times L\times H'}$6 and memory $z_x\in\mathbb R^{D\times L\times H'}$7, whereas the SSM block has time and memory $z_x\in\mathbb R^{D\times L\times H'}$8 (Oshima et al., 2024). For sequences up to 256 frames, the paper reports that SSM-based models “require less memory to achieve the same FVD as attention-based models,” and on UCF101 $z_x\in\mathbb R^{D\times L\times H'}$9 with 16 frames, the FVD values are “272.2” for temporal attention, “286.0” for linear attention, and “226.4” for the SSM model (Oshima et al., 2024). On MineRL with 400 frames, “attention cannot fit in memory, whereas SSM achieves FVD=972.3” (Oshima et al., 2024).

A different temporal reorganization appears in “Temporal Pyramid Video Diffusion Model” (Ran et al., 12 Mar 2025). TPDiff starts from the observation that reverse diffusion is entropy-reducing and that high frame rates are unnecessary in high-entropy stages (Ran et al., 12 Mar 2025). It divides the denoising trajectory into several stages, operating at progressively increasing frame rates, with only the last stage using the full frame rate (Ran et al., 12 Mar 2025). The summary gives the averaged attention cost as

$z_y\in\mathbb R^{D\times L\times W'}$0

which corresponds to an end-to-end speedup of approximately $z_y\in\mathbb R^{D\times L\times W'}$1, though “in practice wall-clock gains of $z_y\in\mathbb R^{D\times L\times W'}$2 are observed” (Ran et al., 12 Mar 2025). The reported training effect is a “50% reduction in training cost,” and the reported inference latencies at 30 denoising steps are “20.8 s → 12.2 s (1.71× faster)” for MiniFlux-vid and “6.0 s → 4.0 s (1.49× faster)” for AnimateDiff (Ran et al., 12 Mar 2025).

Linear-attention transformer backbones extend the same idea to very long and high-resolution generation. “SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer” replaces vanilla attention with a ReLU-kernel linear attention and uses a constant-memory state for block-wise autoregressive generation (Chen et al., 29 Sep 2025). The core attention computation is written as

$z_y\in\mathbb R^{D\times L\times W'}$3

with per-token complexity reduced “from $z_y\in\mathbb R^{D\times L\times W'}$4 (for softmax) to $z_y\in\mathbb R^{D\times L\times W'}$5” in the stated formulation (Chen et al., 29 Sep 2025). The model also introduces a constant-memory block-linear KV cache that stores only two $z_y\in\mathbb R^{D\times L\times W'}$6 matrices and one $z_y\in\mathbb R^{D\times L\times W'}$7 vector, regardless of how many past tokens have been processed (Chen et al., 29 Sep 2025). The summary reports that a 2B-parameter model generates 5 seconds of $z_y\in\mathbb R^{D\times L\times W'}$8 video in “60 s (BF16)” on a single H100 and $z_y\in\mathbb R^{D\times L\times W'}$9 video in “36 s,” while also supporting RTX 5090 deployment with “71 s→29 s (2.4× speedup)” under NVFP4 (Chen et al., 29 Sep 2025). Since this paper was published after the date of some earlier methods, it reflects the continued movement toward long-duration, single-GPU deployment.

4. Step distillation and one-step or few-step generation

Reducing the number of denoising steps is the most direct way to lower inference time. The survey distinguishes consistency distillation, distribution distillation, and adversarial distillation, and states the nominal speed-up as

$L$0

where $L$1 is the original number of denoising steps and $L$2 is the distilled number (Shao et al., 17 Apr 2026). It also emphasizes a quality–NFE trade-off: “lower $L$3 ⇒ higher approximation error and possible temporal drift” (Shao et al., 17 Apr 2026).

“Accelerating Video Diffusion Models via Distribution Matching” distills a pretrained video diffusion model into a four-step student with a denoising GAN discriminator and a 2D score distribution matching loss (Zhu et al., 2024). The student is initialized from the teacher and is trained with a weighted sum of a video-level adversarial distribution matching loss and a frame-level score distribution matching loss (Zhu et al., 2024). The quantitative comparison on four sampling steps reports FVD and CLIPScore values of “1765.30 / 28.60” for Motion Consistency Model, “1405.79 / 28.44” for AnimateLCM, “1623.98 / 29.47” for AnimateDiff-Lightning, “1273.13 / 30.56” for the variant with video SDM, and “1271.45 / 32.01” for the variant with 2D SDM (Zhu et al., 2024). The same summary states that the distilled student “does not require classifier-free guidance at inference” (Zhu et al., 2024).

“AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset” also targets few-step generation, but uses teacher-generated denoising trajectories as a synthetic dataset (Zhang et al., 25 Mar 2025). The student is trained on a reduced set of key timesteps, with the few-step objective

$L$4

combined with an adversarial loss over timestep-conditioned latent distributions (Zhang et al., 25 Mar 2025). The summary reports that the student uses “5 inference steps (versus 50 for teacher)” and that for a “5 s, 720×1280 @24 fps video,” the teacher needs “3234 s” whereas the student needs “380 s – an 8.5× speed-up” (Zhang et al., 25 Mar 2025). At 720P, the reported VBench total scores are “83.24% (teacher) vs 82.77% (ours),” and at 544P “82.67% vs 83.26% (ours slightly better)” (Zhang et al., 25 Mar 2025).

One-step generation represents the most aggressive end of this paradigm. “Mobile Video Diffusion” begins from the Stable Video Diffusion UNet, reduces resolution, introduces temporal multi-scaling, prunes channels and temporal blocks, and then uses adversarial finetuning so that denoising is reduced to a single step (Yahia et al., 2024). The paper reports a progression from “≈45 TFLOPs/step → 1 125 TFLOPs total” for baseline SVD at 25 steps, to “4.34 TFLOPs/clip” after low-resolution finetuning, one-step adversarial distillation, temporal multi-scaling, pruning, and channel funnels (Yahia et al., 2024). The headline efficiency number is “523x more efficient (1817.2 vs. 4.34 TFLOPs),” with “a slight quality drop (FVD 149 vs. 171),” and latent generation for a “14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro” (Yahia et al., 2024).

Single-step diffusion also appears in restoration-oriented settings. “DiffST: Spatiotemporal-Aware Diffusion for Real-World Space-Time Video Super-Resolution” uses a pretrained velocity model and exactly one Euler update,

$L$5

to refine an entire video latent in one pass (Chen et al., 13 May 2026). On 33 frames at $L$6, the summary reports runtime values of “124.29” seconds for VEnhancer, “22.99” seconds for SeedVR2 + BiM-VFI, and “7.12” seconds for DiffST, corresponding to “∼17×” speed-up over the prior single-stage diffusion STVSR baseline (Chen et al., 13 May 2026). Although this setting is super-resolution rather than unconditional generation, it demonstrates that one-step video diffusion can be viable when initialization is already near the target distribution.

5. Compression, pruning, quantization, and architecture search

Model compression methods reduce per-step compute and memory without necessarily changing the diffusion trajectory. The survey divides this class into quantization, pruning, and latent compression, and notes that quantization to 4 bits “cuts MAC cost roughly by $L$7 and memory bandwidth by $L$8 vs. FP32,” while pruning “30–50% of blocks or heads yields 1.3–2× speed-up” (Shao et al., 17 Apr 2026).

“QVD: Post-training Quantization for Video Diffusion Models” is framed as the first PTQ strategy tailored for video diffusion models (Tian et al., 2024). The abstract identifies several quantization-specific obstacles: temporal features exhibit “pronounced skewness,” there are “significant inter-channel disparities and asymmetries in the activation,” and these effects lead to “low coverage of quantization levels by individual channels” (Tian et al., 2024). QVD addresses these issues with High Temporal Discriminability Quantization for temporal features and Scattered Channel Range Integration for improving the coverage of quantization levels across individual channels (Tian et al., 2024). The reported result is “near-lossless performance degradation on W8A8,” outperforming current methods “by 205.12 in FVD” (Tian et al., 2024).

Pruning is treated structurally in “Mobile Video Diffusion” (Yahia et al., 2024). Channel funnels insert trainable low-rank factors between affine layers, which collapse at inference into reduced-rank weights, while temporal block pruning selects only a subset of temporal blocks to execute (Yahia et al., 2024). The paper states that parameter count drops “by ≈30% from channel funnels (½ width) plus 70% of temporal blocks pruned” (Yahia et al., 2024). This is paired with the lossless removal of cross-attention query-key softmax in SVD’s single-key cross-attention, which the summary states “reduces mobile softmax overhead by ∼80%” (Yahia et al., 2024).

Architecture search provides a more global compression mechanism. “SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model” trains a single supernet spanning multiple channel ratios, block-drop patterns, and resolutions (Li et al., 2024). The search space includes dynamic channel ratios $L$9 and fine-grained dropping of convolutional ResBlocks, temporal self-attention, temporal cross-attention, and spatial attention (Li et al., 2024). The reported pixel-space base-model results show a range from “0.64 B” to “1.60 B” parameters, with FVD$512\times1024$0 values “544.4,” “490.5,” “452.2,” and “472.3” for SNED-B, SNED-L, SNED-M, and SNED-S, and A100 inference times “24.4,” “21.2,” “18.1,” and “16.0” seconds respectively (Li et al., 2024). The paper states that “SNED-S (0.64 B) is 1.53× faster than SNED-B with comparable quality” (Li et al., 2024).

These results illustrate a recurrent trade-off. Compression is rarely free: channel reduction, block pruning, quantization, and smaller subnets can preserve or closely approach baseline quality, but the acceptable operating point depends on metric choice, resolution, and deployment target. This suggests that efficient video diffusion models are often families of operating points rather than single optimal architectures.

6. Caching, streamlined inference, and communication-efficient serving

A separate family of methods accelerates inference without retraining, or with minimal architectural intrusion, by reusing features, skipping steps, slicing activations, or partitioning computation across hardware. The survey groups these approaches under cache and trajectory optimization and describes feature caching with gating of the form

$512\times1024$1

where the gate is determined by similarity or by a fixed schedule (Shao et al., 17 Apr 2026).

“Fast and Memory-Efficient Video Diffusion Using Streamlined Inference” is explicitly training-free and combines Feature Slicer, Operator Grouping, and Step Rehash (Zhan et al., 2024). Feature Slicer partitions 5D features by frame or spatial subregions; Operator Grouping ensures that only one slice needs to be active through a group of homogeneous operators; and Step Rehash reuses the temporal layer output of the final up-sampling block for skipped denoising steps (Zhan et al., 2024). The reported peak-memory and latency results are concrete. For AnimateDiff at 16 frames and $512\times1024$2, original inference uses “41.7 GB” and “24.4 s,” whereas the streamlined method uses “11.1 GB” and “15.2 s,” with FVD changing from “758.7” to “784.5” and CLIP score from “28.89” to “28.71” (Zhan et al., 2024). The paper summarizes the overall effect as “3–4× peak memory reduction and 30–40% inference speedup” (Zhan et al., 2024).

“Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Models” uses attention profiling to separate foreground-focused and background-focused blocks, then reuses only background features when inter-step noise similarity is high (Ma et al., 4 Apr 2025). The policy is defined as

$512\times1024$3

The reported Wan2.1 results show baseline latency “497” ms and ProfilingDiT latency “247” ms, corresponding to “2.01×” speedup, with VBench “0.7582” for baseline and “0.7615” for ProfilingDiT; LPIPS, PSNR, SSIM, and FID are also reported as improved over TeaCache-fast (Ma et al., 4 Apr 2025). On HunyuanVideo, ProfilingDiT achieves “1.87×” speedup with latency “932” ms versus “1745” ms for baseline (Ma et al., 4 Apr 2025).

Caching can also be specialized to autoregressive video diffusion. “Efficient Autoregressive Video Diffusion with Dummy Head” identifies that “approximately 25% heads attend almost exclusively to the current frame” and calls them dummy heads (Guo et al., 28 Jan 2026). Dummy Forcing then assigns different heads to sink, neighbor, and dummy memory classes, with dynamic head programming and context packing (Guo et al., 28 Jan 2026). The summary reports “up to 2.0x speedup,” including “24.3 FPS” short-video generation on a single H100, with “less than 0.5% quality drop” (Guo et al., 28 Jan 2026).

When the deployment bottleneck is inter-GPU communication rather than single-device FLOPs, serving strategies dominate. “Communication-Efficient Serving for Video Diffusion Models with Latent Parallelism” partitions the compact latent rather than the model activations and rotates the partition dimension across temporal, height, and width axes (Wu et al., 8 Dec 2025). The communication analysis derives

$512\times1024$4

and for WAN-style models with $512\times1024$5, $512\times1024$6, and $512\times1024$7, reports $512\times1024$8, corresponding to a “$512\times1024$9 reduction” in communication overhead (Wu et al., 8 Dec 2025). On 4 RTX A6000 GPUs, the communication volume for 81 frames is “93 050” MB for naïve model parallelism, “92 690” MB for pipeline parallelism, “7 686” MB for hybrid parallelism, and “2 191” MB for latent parallelism with $\times$0 (Wu et al., 8 Dec 2025). Quality is reported as matching centralized generation within “<0.6% on all benchmarks” (Wu et al., 8 Dec 2025).

Across these methods, the same principle recurs: much of video diffusion inference is redundant because adjacent timesteps, background regions, or local latent partitions change slowly. A plausible implication is that future acceleration will increasingly depend on learned or profiled reliability estimates for reuse, rather than uniform skipping or static schedules alone (Shao et al., 17 Apr 2026).

7. Synthesis, design trade-offs, and open challenges

The literature surveyed here points to a stable taxonomy. Step distillation reduces the number of denoising evaluations; efficient temporal modules reduce the cost of each evaluation; compression reduces arithmetic precision or model width; and cache or trajectory methods avoid recomputing redundant states (Shao et al., 17 Apr 2026). Individual systems frequently combine several of these. MobileVD couples one-step adversarial finetuning with temporal multi-scaling and pruning (Yahia et al., 2024). CMD combines latent decomposition with pretrained image diffusion reuse (Yu et al., 2024). SANA-Video combines linear attention with block-wise autoregression and constant-memory caching (Chen et al., 29 Sep 2025). Streamlined Inference combines slicing, operator grouping, and step reuse without retraining (Zhan et al., 2024).

Several trade-offs recur across papers. First, there is a quality–latency frontier. Lower NFE, more aggressive caching, or heavier pruning can introduce temporal drift, flicker, or loss of fine detail (Shao et al., 17 Apr 2026). Second, method effectiveness depends strongly on deployment regime: long-horizon generation favors linear-time temporal modules or constant-memory caching (Oshima et al., 2024, Chen et al., 29 Sep 2025), mobile deployment favors one-step sampling and operator simplification (Yahia et al., 2024), and distributed serving favors latent-space communication schemes (Wu et al., 8 Dec 2025). Third, temporal consistency is often the failure mode when acceleration is too aggressive, which explains the use of bidirectional SSMs, frame-type embeddings, residual-flow formulations, or background-aware caching to preserve coherent motion (Oshima et al., 2024, Zhou et al., 19 Mar 2026, Salehi et al., 6 Feb 2026, Ma et al., 4 Apr 2025).

Open challenges are identified explicitly in the survey. These include “quality preservation under composite acceleration, hardware-software co-design, robust real-time long-horizon generation, and open infrastructure for standardized evaluation” (Shao et al., 17 Apr 2026). The same survey also calls out “composite acceleration,” “robust streaming,” “long-horizon stability,” “joint routing + kernel design,” and “joint quantization + pruning + sparse attention under a single hardware-aware bit-ops budget” as unresolved directions (Shao et al., 17 Apr 2026). This suggests that the next stage of efficient video diffusion research is likely to be compositional: not a single faster denoiser, but integrated pipelines in which latent representations, temporal operators, caching policies, precision formats, and deployment kernels are optimized together.