---
title: 'Wan2.1 Model: Video Diffusion Transformers'
url: https://www.emergentmind.com/topics/wan2-1-model
type: topic
---

# Wan2.1 Model: Video Diffusion Transformers

The Wan2.1 model family comprises large-scale, transformer-based latent diffusion architectures for high-resolution video generation, editing, and super-resolution. Developed as latent-space Video Diffusion Transformers (DiTs), Wan2.1 models are structured for scalable text-to-video, image-to-video, and video-to-video tasks, featuring multi-head attention over patchified spatio-temporal video representations, frozen text encoders, and adaptive temporal conditioning mechanisms. Model variants are denoted according to parameter count, with Wan2.1-1.3B (1.3 billion parameters) and Wan2.1-14B (14 billion parameters) serving as canonical instances. This platform underpins advanced tasks such as cinematic domain adaptation, affective video synthesis, and 4K video super-resolution, and acts as a reference target for acceleration frameworks such as GalaxyDiT.

## 1. Core Architecture and Model Variants

Wan2.1 implements a transformer-based DiT backbone comprising a sequence of $L$ blocks ($L \approx 40$–$50$), each with a hidden dimension $d \in \{2048, 4096+\}$ and $H \approx 16$–32 attention heads per layer [2512.03451]. The model operates on noised video latents $x_t \in \mathbb{R}^{T P^2 \times d}$ (where $T$ is the number of frames and $P^2$ the number of spatial patches per frame), which are patchified from the outputs of a causal 3D VAE encoder [2507.19138]. The backbone features:

- Multi-head spatio-temporal self-attention and cross-attention to prompt encodings $c$ in every block.
- Conditioning on continuous or learned diffusion timesteps $t$ via AdaLayerNorm (AdaLN).
- Output at each step: a noise estimate $\epsilon_\theta(x_t, c)$, used to iteratively denoise $x_t$ following the DDPM/SDE paradigm.

A frozen large text encoder (e.g., T5-based or Qwen-based) provides prompt embeddings. For video super-resolution and structure-preserving generation, Wan2.1 is often integrated with external modules such as ControlNet CPC (Consistency-Preserved ControlNet), which injects per-frame structural features at each block via residual scaling, with fusion formula:
$$
\mathbf{X}_i^{\mathrm{main}} \leftarrow \mathbf{X}_i^{\mathrm{main}} + \gamma\,\mathbf{F}_{\lfloor i/r\rfloor}^{\mathrm{CPC}}
$$
where $r$ is a block index ratio and $\gamma$ is learned [2507.19138].

Major model variants differ by scale:
| Variant        | Parameters | Hidden Dim. | Use-case Example                |
|----------------|------------|-------------|---------------------------------|
| Wan2.1-1.3B    | 1.3B       | ~2,048      | GalaxyDiT testbed, EmoVid       |
| Wan2.1-14B     | 14B        | ≥4,096      | Cinematic I2V, GalaxyDiT, VSR   |

## 2. Diffusion Process and Conditioning

The forward latent diffusion process employed in Wan2.1 follows:
$$
q(z_t \mid z_0) = \mathcal{N}\left(z_t ; \alpha_t z_0, \sigma_t^2 I \right), \quad z_t = \alpha_t z_0 + \sigma_t \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)
$$
The model learns to invert this process iteratively, using a transformer-based denoiser parameterized as:
$$
\mathbf{v}_\theta(z_t, t) \approx \epsilon - z_0
$$
The network’s reverse denoising step is formulated as:
$$
\hat z_{t-1} = \frac{1}{\alpha_t}(z_t - \sigma_t \mathbf{v}_\theta) + \sigma_{t-1} \epsilon'
$$
where $\epsilon'$ is fresh Gaussian noise for each step.

Text-conditioned generation is achieved via cross-attention blocks, fusing embeddings $c \in \mathbb{R}^{K \times d_c}$ at every layer. In tasks requiring emotion or domain adaptation, this embedding can be augmented by explicit tokens—e.g., emotion word appended to the caption for affective generation, or adapted via Low-Rank Adaptation (LoRA) modules for new domains [2511.11002, 2510.27364].

## 3. Adaptation and Fine-Tuning Methodologies

Wan2.1 supports specialized generative capabilities via lightweight adaptation:

- **Emotion Conditioning:** EmoVid demonstrates that appending an explicit emotion word to the text prompt and applying LoRA modules (rank=32) to cross-attention projections enables emotion-aligned video synthesis without architectural modification [2511.11002]. No new classifier or adversarial losses are introduced; adaptation is driven by the standard diffusion reconstruction objective.
- **Cinematic Domain Transfer:** For cinematic scene synthesis from limited data, LoRA adapters (rank=8, $\alpha$=16) are inserted into cross-attention layers in select ViT encoder and temporal transformer decoder blocks. Adaptation is performed over the visual style (stage 1), followed by inference-time temporal expansion of stylistic keyframes into video sequences (stage 2) [2510.27364].
- **Super-Resolution Integration:** RealisVSR integrates Wan2.1 with ControlNet CPC, injecting per-frame LR features for spatio-temporal consistency, and introduces specialized (wavelet + HOG) high-frequency losses for sharper texture recovery [2507.19138].

Adaptation is typically driven by the standard diffusion loss:
$$
\mathcal{L}_{\mathrm{diffusion}} = \mathbb{E}_{x_0, t, \epsilon} \left\| \epsilon - \epsilon_\theta(x_t, t, c) \right\|_2^2
$$
with optional additional objectives (e.g., temporal consistency loss, total high-frequency rectified loss for VSR).

## 4. Acceleration and Sampling Optimization

GalaxyDiT presents a training-free acceleration framework tailored to Wan2.1, structured around proxy-based step reuse and strict classifier-free guidance (CFG) alignment [2512.03451].

- **CFG in Wan2.1:** Standard classifier-free guidance combines conditional and unconditional model outputs:
  $$
  \text{EG}(x_t, c) = (1 + \gamma) \epsilon_\theta(x_t, c) - \gamma \epsilon_\theta(x_t, \varnothing)
  $$
  Doubling the compute at each diffusion step.

- **Proxy-Based Step Skipping:** GalaxyDiT computes a lightweight proxy (e.g., cross-attention output of the first block) and accumulates a reuse metric per step. The reuse decision is driven by Spearman's $\rho$ correlation with an oracle importance metric, selecting the proxy yielding maximal rank correlation for each model variant (e.g., cross-attention-out for Wan2.1-1.3B with $\rho \approx 0.89$).

- **Guidance-Aligned Reuse:** Both conditional and unconditional passes are co-reused if the reuse metric is below threshold and the step is past the initial 20% iterations, preventing CFG drift artifacts.

This strategy yields up to $2.37\times$ sampling speedup (Wan2.1-14B) with sub-1% fidelity drops on VBench-2.0. PSNR gains surpass prior step-skipping approaches (TeaCache) by $5$–$12$ dB, with notable improvements also in LPIPS and SSIM.

## 5. Application Domains and Benchmarking

Wan2.1’s architecture proves foundational across multiple advanced video generation scenarios:

- **Emotion-Centric Video Generation:** Fine-tuned Wan2.1 on the EmoVid dataset yields significant gains in emotion alignment (EA-8cls: +4–5% absolute), with FVD and CLIPScore improvements in both text-to-video (T2V) and image-to-video (I2V) tasks [2511.11002]. Appended emotion tokens in prompts are sufficient for alignment without auxiliary modules.
- **Cinematic Scene Synthesis:** LoRA-based adaptation enables few-shot learning of historical and cinematic visual domains, preserving spatial style and temporal coherence, with nearly 2$\times$ parallelized inference speedup and minimal perceptual quality loss (LPIPS delta <0.002) [2510.27364].
- **4K Video Super-Resolution:** In RealisVSR, Wan2.1 with CPC, HR-Loss, and text conditioning achieves detail recovery on ultra-high-resolution 4K benchmarks, outperforming prior GAN- and diffusion-based VSR baselines in both quantitative (FVD, PSNR, LPIPS) and qualitative measures [2507.19138].

## 6. Implementation, Training, and Evaluation Details

Key implementation strategies and empirical results for Wan2.1 deployments:

- **Optimization:** AdamW optimizer with learning rates $1 \times 10^{-4}$ [2511.11002] or $3\times 10^{-5}$ [2510.27364], weight decay $\sim0.01$, batch size $1$, epochs $3$–(early stopping), precision bf16, and activation checkpointing as standard practice.
- **Hardware:** 1$\times$ Nvidia H20 or A100 (40–80 GB) shown sufficient for LoRA-based adaptation in hours.
- **Regularization:** Reliance on LoRA adaption limits overfitting, with no adversarial, classifier, or emotion-specific regularizers in reported deployments.
- **Evaluation Metrics:** Fréchet Video Distance (FVD), CLIP-SIM, LPIPS, SSIM, and domain-specific metrics such as EA-2cls/EA-8cls (emotion accuracy) and VBench-2.0 for prompt/video fidelity [2511.11002, 2510.27364, 2512.03451].
- **Software Ecosystem:** PyTorch, DeepSpeed, DiffSynth Studio, OpenCLIP for rapid prototyping and deployment.
- **Pipeline Engineering:** Inference parallelized via temporal sharding and Fully Sharded Data Parallelism (FSDP), with custom overlap blending at shard boundaries to ensure temporal continuity [2510.27364].

## 7. Summary of Benchmarks and Reported Results

Wan2.1 achieves state-of-the-art status as a backbone for a wide spectrum of video generative tasks.

| Task/Metric                | Baseline                 | Wan2.1/Adapted Variant                 | Quantified Outcome                            | Reference      |
|----------------------------|--------------------------|-----------------------------------------|----------------------------------------------|---------------|
| VBench-2.0 fidelity (%)    | 58.36 (Wan2.1-14B base)  | 57.64 (GalaxyDiT-fast, 2.37$\times$)    | $-0.72\%$ delta, $+12$dB PSNR, $+27\%$ SSIM  | [2512.03451]  |
| Emotion (EA-8cls, T2V)     | 44.16 (before)           | 48.33 (after EmoVid fine-tune)          | $+4.2\%$ absolute                            | [2511.11002]  |
| I2V LPIPS                  | 0.0325 (base)            | 0.0324 (after EmoVid fine-tune)         | minimal impact on perceptual similarity       | [2511.11002]  |
| Cinematic LPIPS            | 0.142                    | $<$0.002 degradation after sharded inf. | expert user $+1.2$ mean rating (p$<$0.05)    | [2510.27364]  |

The Wan2.1 model family demonstrates high scalability, modularity for conditional adaptation, and competitive efficiency when paired with modern acceleration techniques. Its transformer-based block design and latent video diffusion framework form a technical foundation for high-fidelity, flexible, and rapid video generative modeling across diverse domains.

Source: https://www.emergentmind.com/topics/wan2-1-model