Lightweight Causal ConvNet Decoder for Video Synthesis

• The paper introduces Flash-VAED which employs independence-aware channel pruning, operator replacement, and a dynamic distillation framework to optimize video VAE decoders.
• It utilizes causal 3D convolutions to enforce temporal consistency, while replacing heavy operations with efficient depthwise and spatial convolutions in deep and shallow stages.
• Extensive evaluations show over 5× speedup with minimal degradation in reconstruction quality, enabling plug-and-play integration into diffusion-based video synthesis pipelines.

A lightweight causal ConvNet decoder refers to an optimized neural network component for video generation that employs temporal causality and architectural pruning for fast inference, minimizing redundancy and computational cost while retaining high-fidelity outputs. Flash-VAED ("Flash Variational AutoEncoder Decoder") exemplifies this approach, introducing a universal acceleration pipeline for VAE (Variational Autoencoder) decoders in diffusion-based video synthesis. By integrating independence-aware channel pruning, operator replacement, and dynamic distillation, the lightweight causal ConvNet decoder achieves substantial speedups and preserves reconstruction quality, enabling plug-and-play compatibility with existing diffusion model pipelines (Zhu et al., 22 Feb 2026).

1. Decoder Architecture and Causal 3D Convolutions

The foundational architecture for the lightweight causal ConvNet decoder is the video VAE decoder, which processes a latent tensor $z\in\mathbb{R}^{C\times T'\times H'\times W'}$ through a sequence of resolution-changing blocks: a "mid" block at lowest resolution, followed by a series of upsampling blocks $\mathrm{up}_0,\ldots,\mathrm{up}_3$, each doubling or rescaling the spatiotemporal dimensions to synthesize full-size video. Within each block, causal 3D convolutions (CausalConv3D) enforce temporal causality by restricting predictions at time $t$ to depend only on past and present inputs ($\tau\le t$).

A CausalConv3D is formally specified by: $$y_{c_{\rm out},\,t,h,w} \;=\; \sum_{c_{\rm in}=1}^{C_{\rm in}} \sum_{\tau=0}^{k_T-1} \sum_{i=-\lfloor k_H/2\rfloor}^{\lfloor k_H/2\rfloor} \sum_{j=-\lfloor k_W/2\rfloor}^{\lfloor k_W/2\rfloor} K_{c_{\rm out},c_{\rm in},\,\tau,\,i+\lfloor k_H/2\rfloor,\,j+\lfloor k_W/2\rfloor} \;x_{c_{\rm in},\,t-\tau,\,h+i,\,w+j}$$ with $\tau\ge0$. The resulting operation is the main contributor to inference latency (60–80% per block), with the FLOP count: $$\mathrm{FLOPs}_{\rm CausalConv3D} = T\,H\,W \times C_{\rm in}\,C_{\rm out} \times k_T\,k_H\,k_W$$ Implementing causal convolutions in all upsampling stages ensures strict autoregressive order, essential for high-fidelity video synthesis.

2. Independence-Aware Channel Pruning

Severe channel redundancy in state-of-the-art video VAEs is addressed using comprehensive channel pruning mechanisms. Unlike conventional pairwise cosine similarity-based pruning, Flash-VAED designates a channel as redundant if it can be (approximately) linearly reconstructed by a small subset of retained channels. Given feature maps $\mathbf Y \in \mathbb{R}^{C\times N}$ (where $N=THW$), a subset $\mathbf X\in\mathbb{R}^{r\times N}$ is selected and a projection is fit: $\mathrm{up}_0,\ldots,\mathrm{up}_3$0 The coefficient of determination,

$\mathrm{up}_0,\ldots,\mathrm{up}_3$1

quantifies reconstruction quality. Empirical SVD shows that approximately 22% of channels suffice to explain 99% of the variance.

Flash-VAED pruning is executed via:

1. Greedy channel selection: iteratively selecting channels to maximize marginal gain in $\mathrm{up}_0,\ldots,\mathrm{up}_3$2 until $\mathrm{up}_0,\ldots,\mathrm{up}_3$3 are retained.
2. Expressivity enhancement: prior to pruning, jointly minimizing

$\mathrm{up}_0,\ldots,\mathrm{up}_3$4

with gradients masked to update only retained-channel filters.

1. Shortcut injection: replacing any identity shortcut in a residual block with a $\mathrm{up}_0,\ldots,\mathrm{up}_3$5 conv initialized by $\mathrm{up}_0,\ldots,\mathrm{up}_3$6, preserving topology when consecutive blocks utilize differing channel subsets.

3. Stage-Wise Operator Replacement: Dominant Operator Optimization

To further reduce inference latency, the dominant operator, CausalConv3D, is systematically replaced in a stage-wise fashion by less expensive convolutions:

• Deep stages ($\mathrm{up}_0,\ldots,\mathrm{up}_3$7): CausalConv3D is factorized into depthwise-separable 3D convolution and a pointwise $\mathrm{up}_0,\ldots,\mathrm{up}_3$8 convolution, substantially lowering FLOPs by an approximate factor of $\mathrm{up}_0,\ldots,\mathrm{up}_3$9.
• Shallow stages ($t$0): CausalConv3D is replaced by spatial-only 2D convolutions (with kernel size $t$1), operating on each frame independently. Temporal causality is delegated to the earlier, lower-resolution layers without empirical loss of output quality.

The replacement pipeline: $\tau\le t$1 This hybridization of depthwise and 2D spatial convolutions is central to the observed efficiency gains; substituting 3D with 2D convolutions in shallow layers yields a 4–5× reduction in FLOPs with negligible (<0.3 dB) PSNR loss.

4. Three-Phase Dynamic Distillation Framework

To transfer the full latent-to-video synthesis capability from the original decoder to the lightweight version, Flash-VAED utilizes a feature-based dynamic distillation scheme over three phases. For each block $t$2, feature maps from the original ($t$3) and Flash-VAED ($t$4) decoders are compared using: $t$5

Phases are:

• Phase 1: Global Alignment The objective

$t$6

aligns high-level feature distributions.

• Phase 2: Channel Expressivity Addition of expressivity loss $t$7 encourages the pruned network to maintain reconstruction power:

$t$8

• Phase 3: Shallow Layer & Projection Distillation Employing a $t$9 conv initialized by the OLS matrix $\tau\le t$0 (from channel pruning), the last phase aligns shallow, pruned layers without sacrificing spatial detail.

5. Performance Evaluation and Ablations

Extensive benchmarking on Wan and LTX-Video VAE decoders confirms the effectiveness of the lightweight causal ConvNet decoder:

Scenario	Baseline FPS	Flash-VAED FPS / Speedup	Fidelity Metrics / Drop
Wan 2.1, RTX 5090D	19.3	118.8 (6.16×)	93.1% PSNR, SSIM 0.9614
Jetson Orin	0.65	3.70 (5.7×)
LTX-Video (¼ channels)	204	1,168	-0.93% PSNR drop

• End-to-end pipeline improvements:
  • Self-Forcing Wan 1.3B: 27% latency reduction
  • FastVideo Wan 1.3B: 36% reduction
• Reconstruction quality:
  • Wan 2.1: PSNR 37.61 dB
  • LTX-Video: 96.9% PSNR, SSIM 0.9293, LPIPS 0.0551
• Ablation results:
  • Pruning only ¼ channels yields >5× speedup with <1% PSNR loss.
  • Operator replacement (3D to 2D in shallow blocks) achieves ~5× FLOP reduction and <0.3 dB loss.
  • Use of all three distillation phases results in PSNR 32.24 dB; omitting any phase significantly degrades quality (30.8–31.2 dB).

These metrics establish that the lightweight causal ConvNet decoder design achieves robust acceleration with minimal fidelity compromise, supporting efficient plug-and-play deployment in contemporary video diffusion architectures.

6. Applications and Integration in Latent Diffusion Video Models

Lightweight causal ConvNet decoders are integral to modern latent diffusion pipelines for video generation, where the bottleneck has shifted from transformer backbones to the VAE decoder. Flash-VAED provides universal compatibility; its methodology can be directly integrated into any video VAE decoder, replacing the pre-existing decoder without retraining or data format modification. Empirical evaluations on VBench-2.0 support its zero-modification deployment and highlight its effectiveness for both server-class GPUs and edge inference hardware.

A plausible implication is that, as diffusion transformers become more efficient, further research focus will shift toward lightweight decoders to balance overall system latency and resource consumption. This suggests ongoing relevance and continued development of operator replacement, structured pruning, and distillation protocols for scalable, high-temporal-resolution video synthesis (Zhu et al., 22 Feb 2026).