SkyReels V4: Unified Multimedia Generation

• SkyReels V4 is a unified multi-modal video foundation model that supports joint video-audio generation, inpainting, and editing using a dual-stream transformer architecture.
• It employs a Multimodal Diffusion Transformer to achieve precise temporal alignment and semantic coherence across diverse inputs such as text, images, video clips, masks, and audio.
• The model integrates efficient low-to-high resolution synthesis and a multi-stage training protocol, delivering state-of-the-art results in high-resolution audiovisual content creation.

SkyReels V4 is a unified multi-modal video foundation model designed for joint video-audio generation, inpainting, and editing. It employs a dual-stream Multimodal Diffusion Transformer (MMDiT) architecture, where distinct branches synthesize temporally aligned video and audio sequences while sharing a common Multimodal LLM (MMLM) for prompt and instruction encoding. SkyReels V4 supports a broad range of input modalities—including text, images, video clips, masks, and audio references—to enable flexible, high-fidelity audiovisual content generation and transformation, up to 1080p resolution, 32 frames per second, and 15-second duration, with strong computational efficiency and semantic alignment (Chen et al., 25 Feb 2026).

1. Dual-Stream Multimodal Diffusion Transformer Architecture

SkyReels V4 utilizes a backbone architecture comprising twin transformer branches (Video MMDiT and Audio MMDiT) with shared configuration and a frozen MMLM text encoder for rich, multi-modal prompt conditioning. The initial M layers of both branches follow a dual-stream structure, with video, audio, and text tokens maintaining separate normalization and linear projections, while participating in joint self-attention to facilitate early cross-modal alignment. The subsequent N layers merge into a single-stream regime: all token types are processed with shared transformer parameters, with cross-attention to text retained for the video branch to reinforce semantic coherence.

Block Computation Flow

$t\sim \text{Uniform}(0,1)$2

Within each Video MMDiT block, the computation proceeds through joint attention among video, text, and audio tokens, cross-attention mechanisms for video-to-audio information transfer, and, in later layers, single-stream attention for parameter efficiency:

$t\sim \text{Uniform}(0,1)$3

This design enables strong early fusion of semantics and tightly coupled video-audio generation.

2. Diffusion and Inpainting: Mathematical Principles

The generation process follows a flow-matching diffusion paradigm over continuous-time latent trajectories for both video and audio. For latent representations $z^0_v$ (video) and $z^0_a$ (audio):

• A noise time $t\sim \text{Uniform}(0,1)$ and noise vectors $\epsilon_v, \epsilon_a \sim \mathcal{N}(0, I)$ are sampled.
• Noisy latents are formed:

$$z^t_v = t z^0_v + (1-t) \epsilon_v,\quad z^t_a = t z^0_a + (1-t) \epsilon_a$$

• The model predicts velocity fields $v^{v}_\theta$, $v^{a}_\theta$ to match underlying data distributions via the loss:

$$\mathcal{L} = \mathbb{E}_{t,z^0,\epsilon}\Big[\|v^{v}_\theta(t,z^t_v,z^t_a,c)-(z^0_v-\epsilon_v)\|^2 + \|v^{a}_\theta(t,z^t_a,z^t_v,c)-(z^0_a-\epsilon_a)\|^2\Big]$$

Sampling is performed by backward Euler integration:

$t\sim \text{Uniform}(0,1)$4 where $γ(t) = dt / (1-t)$.

For inpainting and editing, all video tasks—including text-to-video (T2V), image-to-video (I2V), extension, inpainting, and editing—are unified as masked inpainting with channel concatenation:

$$Z_{\text{input}} = \mathrm{Concat}(V, I, M) \in \mathbb{R}^{T \times H \times W \times (2C+1)},$$

where $z^0_a$0 is the noisy video, $z^0_a$1 the conditional frame latents (masked where absent), and $z^0_a$2 a binary mask. Specific mask patterns encode T2V, I2V, temporal extension, and spatial editing.

3. Instruction Modality and In-Context Learning

The architecture is designed to follow complex, multi-modal instructions. All input modalities—free-form text, reference images, video clips, masks, and audio snippets—are concatenated and embedded by the frozen MMLM.

Visual in-context conditioning is achieved by prepending condition latents $z^0_a$3 (extracted via VAE) to the video latent tensor $z^0_a$4 and assigning negative temporal indices through 3D rotary position encoding (RoPE):

$z^0_a$5

The concatenated conditioning allows the model to directly propagate fine-grained patterns or scene context into the generative process through self-attention, enabling instruction following such as object insertion or region-specific edits. Reference audio snippets are similarly encoded into audio latents with RoPE offset and made available to the Audio MMDiT branch, with bi-directional cross-attention mechanisms allowing tightly coupled audio-video synchronization:

$z^0_a$6

This joint conditioning enables highly granular, context-aware video and audio generation, supporting advanced use cases such as dialog-driven animation, vision-referenced inpainting, and prompt-based scene manipulation.

4. Computational and Memory Efficiency: 1080p, 32 FPS, 15s

To enable tractability at cinematic scales, SkyReels V4 employs a multi-resolution synthesis strategy:

• The base model predicts all frames at low spatial resolution ($z^0_a$7) and select keyframes (e.g., every 4th) at high resolution ($z^0_a$8p).
• A Refiner module linearly upsamples low-res latents in space and time, replaces keyframe slots with high-res predictions, and passes the hybrid latent sequence through a DiT (Diffusion Transformer) with Video Sparse Attention for super-resolution and temporal interpolation.

The frame interpolation and super-resolution process is formalized as:

$t\sim \text{Uniform}(0,1)$5

Video Sparse Attention (VSA) dramatically reduces attention cost: for high-res inputs, spatio-temporal cubes are pooled, top-K regions selected, and dense attention applied only within those. This yields approximately 3× reduction in attention memory/computation with negligible quality loss.

Table: Efficiency features and specifications

Aspect	Details	Resource Cost
Max Resolution	$z^0_a$9 @ 32 FPS × 15 sec	≈480 frames
Wall-clock Time	<$t\sim \text{Uniform}(0,1)$0 min on $t\sim \text{Uniform}(0,1)$1A100	For 15s generation
Memory Peak	≈80 GB (with temporal block splitting, VSA)

5. Training Protocol and Evaluation

SkyReels V4 is trained in a multi-stage schedule:

1. Video Pretrain: Six phases incrementally build up from text-to-image, low-res text-to-video, inpainting, mixed resolutions, high-res, and multi-modal conditioning, using 3 billion images and 0.5–1 billion videos.
2. Audio Pretrain: Hundreds of thousands of hours of speech, music, and SFX (≤ 15s).
3. Joint Video-Audio Pretrain: Mixes T2V, T2AV, and T2A objectives, with approximately 50% data from prior video pretrain.
4. Supervised Fine-Tuning: 5 million multi-modal videos, followed by 1 million curated high-quality examples.

Loss balancing is conducted by summing the video and audio flow-matching losses equally, with optional scaling by the square root of latent size to normalize gradient contributions.

Evaluation Metrics and Results

• Artificial Analysis Arena (public Elo): Rank 3/10 on text-to-video+audio track.
• SkyReels-VABench (n>2,000 prompts, human scoring):

| Metric | SkyReels V4 Score | |-------------------------|---------------------------| | Instruction Following | 4.3/5 (best) | | Motion Quality | 4.2/5 (best) | | Visual Quality | 4.0/5 (tied best) | | Audio-Video Sync | 3.9/5 (top quartile) | | Audio Quality | 3.8/5 |

• Good-Same-Bad pairwise: Preferred as “Good” 60–70% of time vs. top-tier commercial baselines.
• Automatic lip-sync offset (<3 frames): >95% pass rate.
• Inpainting PSNR/SSIM on held-out mask tasks: surpasses Unified-former and CogVideoX by +1.2dB / +0.04 SSIM on average.

6. Applications and Significance

SkyReels V4 provides a unified, end-to-end system for cinematic audiovisual content creation, subsuming text-to-movie generation, reference-driven animation, multi-shot storytelling, inpainting, and precise video-audio editing with strong temporal alignment. Its architectural innovations—including dual-stream early-fusion, channel-concatenation inpainting, in-context instruction following, and joint low/high-res synthesis—represent a comprehensive advance in controllable, high-resolution multi-modal generation (Chen et al., 25 Feb 2026).