Wan-Alpha: Deep RGBA Video Synthesis

• Wan-Alpha is a deep generative framework that synthesizes high-fidelity RGBA videos with integrated transparency for seamless post-production.
• It employs a specialized VAE with LoRA adapters and a diffusion transformer, jointly modeling RGB and alpha channels for improved motion coherence.
• The framework demonstrates significant improvements in temporal stability and edge clarity, addressing limitations of previous text-to-video synthesis models.

Wan-Alpha is a deep generative framework for high-fidelity text-to-video synthesis with explicit support for transparency via the alpha channel. Addressing limitations in previous approaches—such as poor motion coherence, jagged alpha edges, and the inability to render semi-transparent objects or fine-grained details—Wan-Alpha jointly models RGB and alpha information, enabling the creation of RGBA videos suitable for direct compositing in media production, game engines, visual effects, and digital asset creation. The system features a variational autoencoder (VAE) tailored for RGBA fusion, a text-conditioned diffusion transformer, and a carefully constructed RGBA video-caption dataset supporting high-quality, efficient generation (Dong et al., 29 Sep 2025).

1. Motivation and Problem Scope

RGBA video generation aims to synthesize videos where each frame comprises four channels: red, green, blue, and alpha (transparency). Pixel-wise alpha enables downstream compositing—that is, seamless integration of the generated content into other visual assets or backgrounds. The practical importance of alpha-aware video is established across video editing, animation, VFX, AR/VR asset creation, and game content pipelines.

Despite its significance, generating RGBA video is encumbered by three challenges:

• Data scarcity and expense: RGBA-annotated video datasets are far rarer and costlier to produce than standard RGB datasets.
• Technical limitations of prior methods: Existing text-to-video models ignore transparency or use image-level matting techniques without temporal consistency, requiring excessively large RGBA datasets (e.g., LayerDiffuse's 1M images), or duplicating network architectures (e.g., TransPixeler), resulting in computational overhead and degraded quality.
• Suboptimal synthesis outcomes: Prior work exhibits low visual quality, poor temporal and motion coherence, jagged or aliased alpha boundaries, and failure to represent fine semi-transparent details or visual phenomena such as glowing, flickering, or intricate edge structures.

Wan-Alpha’s strategy is to jointly learn RGB and alpha synthesis within a unified generative modeling pipeline, thereby resolving these bottlenecks and advancing the state of the art in transparent video synthesis (Dong et al., 29 Sep 2025).

2. Architecture and Objective Functions

2.1 RGBA-aware Variational Autoencoder

Wan-Alpha’s VAE architecture extends a pretrained RGB video VAE (Wan-VAE) by "copying" it for the alpha channel and introducing a feature fusion mechanism. The process comprises:

• Hard rendering for disambiguation: Input RGBA videos $V=(V_{rgb}, V_\alpha)$ are rendered with random background color $\bar c$ to avoid confusion between colored backgrounds and transparency:

$$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$

where $\mathbb{R}^h$ is a hard rendering operator mapping $\alpha > 0 \mapsto 1$, $\alpha = 0 \mapsto 0$.

• Joint encoding and merging: Both $\bar V_{rgb}$ and $V_\alpha$ are encoded via a frozen encoder $\mathcal{E}$, concatenated, and passed to a learnable causal 3D conv + attention feature merge block $\mathcal{M}$:

$\bar c$0

yielding spatiotemporal latent representations.

• Decoding with LoRA adaptation: Two decoders (RGB and alpha) are augmented with low-rank adapters (LoRA, rank 128), with both branches independently reconstructing the video channels:

$\bar c$1

2.2 VAE Losses

Supervision is structured across three output targets:

• Soft-rendered RGB: $\bar c$2 using soft-compositing.
• Hard-rendered RGB: $\bar c$3 using discrete alpha thresholds.
• Raw Alpha: $\bar c$4.

The total VAE loss aggregates reconstruction ($\bar c$5), VGG perceptual (feature MSE), and Sobel edge losses for boundary sharpness: $\bar c$6

$\bar c$7

No KL divergence term is imposed at the merge block, since latents are predicted directly atop the pretrained VAE.

2.3 Diffusion Transformer

Latents $\bar c$8 are input to a video diffusion transformer (pretrained Wan model) with self-attention over space-time and cross-attention to text encodings (umT5 encoder). Each transformer layer integrates DoRA (weight-decomposed low-rank adapter, rank 32) for lightweight fine-tuning. The diffusion process follows the Rectified Flow framework: $\bar c$9 Inference uses 4 LightX2V-accelerated sampling steps, generating 81 frames at $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$0 in approximately 128 seconds on standard GPU hardware.

3. Dataset Construction

Wan-Alpha addresses RGBA video data scarcity by aggregating and synthesizing diverse multi-source datasets.

VAE Training Data

• Sources: 10 matting image datasets, including AIM-500, AM-2K, Distinctions-646, HHM2K, HHM50K, Human2K, P3M-10K, RealWorldPortrait-636, SIMD, Transparent-460 (converted to short video snippets), and multiple video matting datasets (DVM, VideoMatting108, VideoMatte240K) plus select Internet clips.
• Splits: 77,237 training and 4,066 validation videos, each 17 frames at $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$1.

Text-to-Video RGBA Data

• Composition: 437 total video/caption pairs (301 video, 20 image, 116 special-effects) with explicit transparent backgrounds.
• Annotation: Captions are produced by Qwen2.5-VL-72B, then human-refined (in Chinese) with attribute tags (motion speed, style, shot size, quality) for semantic richness.
• Format: 4-5s clips at 16 FPS, resolution $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$2.

Editor's term: "Wan-Alpha RGBA dataset" refers to this combined data resource.

4. Training Procedure

VAE Training

• Iterations: 60,000
• Batch size: 2
• LoRA rank: 128 per decoder
• Augmentation: Random resize and crop

Diffusion Transformer Training

• Iterations: 1,500
• Batch size: 8
• DoRA rank: 32
• Backbone: Wan video transformer (structure unchanged except for DoRA insertion)
• Prompt extension: Mirrors Wan implementation; no classifier-free guidance at inference

The optimizer is likely AdamW with learning rate ≈1e-4 (optimizer details not specified).

5. Experimental Analysis

Qualitative Results

Comparative evaluation is provided against two TransPixeler variants (open-sourced on CogVideoX-5B and Adobe Firefly). Test prompts span a wide range (e.g., flying parrot, spinning coin, candle flame, special-effect overlays). Wan-Alpha demonstrates:

• Sharper, temporally stable alpha edges and silhouettes;
• Faithful rendering of semi-transparent phenomena (feathers, flames, glass, bubbles);
• Fine structural detail (hair strands, condensation, ink overlays);
• Targeted effects (soft glows, light scattering, transparent overlays).

Figures 10 (parrot) and 17 (hair) specifically highlight performance improvements.

Quantitative and Efficiency Assessment

• Frame and speed comparison:
  • TransPixeler (open): 49 frames at $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$3, 8 FPS, ≈32 min.
  • Wan-Alpha: 81 frames at $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$4, 16 FPS, 128 s (∼15× faster).
• Wan-Alpha yields a significant speedup while improving temporal and spatial quality.

No FID/LPIPS or human user study scores are provided; the main evidence comprises runtime, resolution, and qualitative frame comparisons.

6. Ablations and Observations

While no standalone ablation section is provided, the authors report key findings during development:

• Omitting the Sobel edge loss ($$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$5) produces blurred alpha contours.
• Removing the feature merge block $$\bar V_{rgb} = \mathbb{R}^h(V_{rgb},V_\alpha,\bar c)$$6 (i.e., direct tensor concatenation) leads to coupling errors between color and transparency, resulting in compositional confusion.
• Forgoing manual caption refinement in text–video data impairs semantic correspondence and output relevance.

A plausible implication is that joint modeling techniques and explicit edge regularization are essential for temporally stable and visually plausible RGBA video synthesis.

7. Limitations and Future Research

Wan-Alpha’s main failure modes, as discussed in Figures 19–20, include:

• Occasional errant background debris or unintended artifacts (notably in effects-heavy scenes).
• Incomplete or over-smoothed alpha masks, causing missing holes or inaccurate object boundaries.

Outlined future directions include:

• Incorporating explicit temporal alpha consistency regularization;
• Scaling to broader, more complex RGBA video corpora;
• Integrating stronger object-centric or physics-driven priors to minimize compositional artifacts;
• Further public releases with ongoing quality refinement (Dong et al., 29 Sep 2025).