Wan-S2V: Audio-Driven Video Synthesis

• Wan-S2V is an audio-driven video generation model that produces film-quality character animation by integrating audio, text, and visual cues.
• It combines a 3D-VAE encoder/decoder, Wav2Vec-based audio encoding, and FramePack compression within a diffusion framework to achieve nuanced motion and dynamic camera work.
• The model demonstrates state-of-the-art performance in metrics like FID, SSIM, and PSNR, making it ideal for complex film and TV production applications.

Wan-S2V is an audio-driven video generation model designed to produce film-quality character animation with high expressiveness, fidelity, and cinematic realism. Building on the Wan DiT-based video diffusion backbone, Wan-S2V integrates multimodal conditioning with novel architectural components for nuanced character interactions, realistic motion, and dynamic camera work. The system targets scenarios demanding greater complexity than speech or singing animation, such as film and television production, by combining audio, reference imagery, text prompts, and extended motion histories using dedicated token compression and cross-modal attention mechanisms (Gao et al., 26 Aug 2025).

1. Architecture and Pipeline

Wan-S2V extends the Wan video diffusion backbone with several key modules:

• 3D-VAE Encoder/Decoder: RGB video frames $X\in\mathbb R^{F\times H\times W\times 3}$ are encoded to latent space $x_0=\mathcal E_{\rm VAE}(X)$, with $x_0\in\mathbb R^{f\times h\times w\times c}$. At each diffusion timestep $t$, noise is added via Flow Matching, $x_t = t\epsilon + (1-t)x_0$, and the UNet backbone predicts the velocity $$v_\theta(x_t, t) \approx \frac{dx}{dt} = \epsilon - x_0$$.
• Audio Encoder and Audio Blocks: The model extracts features from raw 16kHz waveforms using a Wav2Vec encoder and processes these into per-frame audio tokens $a_i\in\mathbb R^{f\times t\times c}$ through causal 1D convolutions. In each Audio Block, visual latents $x_t$ are reshaped and attend to corresponding $a_i$ tokens using segment-wise cross-attention, avoiding full 3D attention overhead.
• FramePack Compressor: This module compresses long histories of motion latents $\{M_j\}$ by down-sampling earlier frames more heavily, allowing storage and inference over hundreds of frames at constant token budget, critical for long-form scene consistency.
• Text Encoder: Frozen from Wan, natural language scene/camera/action prompts are encoded and injected by cross-attention in the diffusion backbone.
• Inference: Starting from Gaussian noise $x_0=\mathcal E_{\rm VAE}(X)$0, the learned $x_0=\mathcal E_{\rm VAE}(X)$1 iteratively recovers $x_0=\mathcal E_{\rm VAE}(X)$2 conditioned on all inputs, which is then decoded to video frames by the 3D-VAE decoder.

2. Input/Output Modalities

Wan-S2V's multimodal conditioning supports the following:

• Audio Input: Raw 16kHz waveform processed by Wav2Vec to multi-level features, with shallow layers encoding prosody/emotion and deeper layers lexical content. Temporal alignment achieved via 1D convs ensures frame-accurate audio-visual correlation.
• Video Input: A single or small set of reference images at user-defined resolution for identity/appearance; optional motion frame histories for continuity, compressed by FramePack; and a text prompt specifying scene-level and cinematic camera context.
• Video Output: Generates up to 48 RGB frames per sequence, at resolutions up to $x_0=\mathcal E_{\rm VAE}(X)$3. Outputs maintain reference identity, follow high-level text/camera instructions, and synchronize character motion tightly with the supplied audio.

3. Training Methodology

• Loss Function: Wan-S2V is trained end-to-end with the flow-matching regression objective:

$x_0=\mathcal E_{\rm VAE}(X)$4

No adversarial, explicit reconstruction, or separate lip-sync losses are employed. All conditioning information is delivered through concatenation or attention in the DiT backbone.

• Multi-Stage Curriculum:

1. Audio encoder pre-training on speech and singing videos
2. Full model pre-training on diverse videos (speech + cinema/TV)
3. Supervised fine-tuning on high-quality film clips (curated manually)

• Training Infrastructure: Model parallelism leverages FSDP over 14B parameters on 8 GPUs, with RingAttention and Ulysses enabling near-linear context parallel speedup. Training iterations are reduced to ~12s on 8 GPUs. Variable-resolution video training is managed by a global max token budget $x_0=\mathcal E_{\rm VAE}(X)$5, with FramePack tokens for long sequences.
• Resolution and Length: Sequences up to 48 frames at $x_0=\mathcal E_{\rm VAE}(X)$6 are supported in training and inference.

4. Experimental Evaluation

• Metrics Used:
  • Per-frame: FID (Fréchet Inception Distance), SSIM, PSNR
  • Video-level: FVD (Fréchet Video Distance), Sync-C (lip-sync), CSIM (facial identity), HKC/HKV (hand quality), EFID (expression divergence)
• Benchmark Results:

| Method | FID↓ | FVD↓ | SSIM↑ | PSNR↑ | Sync-C↑ | EFID↓ | HKC↑ | HKV↑ | CSIM↑ | |-----------------|-------|--------|-------|-------|---------|-------|------|------|-------| | EchoMimicV2 | 33.42 | 217.71 | 0.662 | 18.17 | 4.44 | 1.052 | 0.425|0.150 | 0.519 | | MimicMotion | 25.38 | 248.95 | 0.585 | 17.15 | 2.68 | 0.617 | 0.356|0.169 | 0.608 | | EMO2 | 27.28 | 129.41 | 0.662 | 17.75 | 4.58 | 0.218 | 0.553|0.198 | 0.650 | | FantasyTalking | 22.60 | 178.12 | 0.703 | 19.63 | 3.00 | 0.366 | 0.281|0.087 | 0.626 | | Hunyuan-Avatar | 18.07 | 145.77 | 0.670 | 18.16 | 4.71 | 0.708 | 0.379|0.145 | 0.583 | | Ours (Wan-S2V) | 15.66 | 129.57 | 0.734 | 20.49 | 4.51 | 0.283 | 0.435|0.142 | 0.677 |

Wan-S2V achieves state-of-the-art FID, SSIM, PSNR, and CSIM; FVD is on-par with EMO2. Hand motion scores (HKC, HKV) are competitive, despite not using explicit MANO hand-model control. Qualitative evaluation highlights robust identity preservation under large head/body poses (vs. Hunyuan-Avatar), larger motion amplitude (vs. OmniHuman), and superior cross-clip consistency and prop tracking due to FramePack.

5. Ablation and Model Analysis

No formal ablation table is present, but the source reports the following qualitative insights:

• Full-Parameter Fine-Tuning: Outperforms partial tuning by resolving conflicts between text and audio modalities.
• FramePack Compression: Enables extended motion-frame context, doubling effective history. Removing FramePack halves history length and degrades visual consistency across scene cuts and segments.
• Training Curriculum: Stabilizes audio conditioning. A plausible implication is that sequential curriculum on audio, general video, and hand-picked film material is critical for achieving cinematic fidelity and robust multi-modal association.

6. Applications and Extensions

Wan-S2V enables use cases beyond conventional talking-head animation:

• Long-Form Video Generation: FramePack's variable compression ratios allow for coherent scenes extending to nearly a minute without quadratic increase in computational demands.
• Precise Lip-Sync Editing: The system achieves Sync-C ≈ 4.5 without auxiliary networks, enabling frame-accurate re-dubbing or audio swap-in for post-production filmmaking.
• Cross-Segment Prop and Camera Consistency: The model maintains item and motion coherence across scene cuts, addressing a key requirement for in-production film use.

7. Future Directions

Future research directions outlined include:

• Audio-Driven Multi-Person Interaction: Extending capability to coordinated group or crowd scenes under purely audio cues.
• Dynamic Camera Choreography: Enabling camera motion and framing dictated directly by audio modulation and temporal effects.
• Broader Domains: Scaling to genres such as dance, theater, or animation of non-human characters to widen the scope of audio-driven cinematic synthesis (Gao et al., 26 Aug 2025).

Wan-S2V constitutes the inaugural entry in the “Vida” series of models aimed at bridging local (audio) and global (text/camera) control for high-fidelity, generative film/video synthesis.