---
title: 'SkyReels-V4: Unified Multi-Modal Generation'
url: https://www.emergentmind.com/topics/skyreels-v4
type: topic
---

# SkyReels-V4: Unified Multi-Modal Generation

SkyReels-V4 is a unified multi-modal video foundation model designed for joint video-audio generation, inpainting, and editing at cinematic resolutions and durations. The model introduces a dual-stream Multimodal Diffusion Transformer (MMDiT) architecture, with one branch dedicated to synthesizing video and the other to generating temporally aligned audio. These streams are unified by a shared text encoder based on a frozen Multimodal Large Language Model (MMLM), enabling the flexible conditioning of generation on rich multi-modal instructions including text, images, video clips, audio, and masks. SkyReels-V4 supports a spectrum of video and audio tasks, handling resolutions up to 1080p, frame rates of 32 FPS, and sequences of up to 15 seconds, while also achieving computational efficiency through a joint low-resolution and keyframe-based generation strategy, subsequently refined by super-resolution and interpolation modules [2602.21818].

## 1. Dual-Stream Multimodal Diffusion Transformer Architecture

The core architectural innovation in SkyReels-V4 is the dual-stream MMDiT, featuring two parallel Transformer-based diffusion networks of identical embedding dimensions: one for video latents and one for audio latents. The video branch is initialized from a pretrained DiT-style text-to-video model, while the audio branch is trained from scratch with an analogous architecture. At each of the initial M layers, the model maintains separate QKV projections and layer normalization/MLP parameters for the video or audio stream ($x_v$ or $x_a$) and the shared text stream $x_t$, but performs joint self-attention across both:

\[
\begin{aligned}
Q_v,K_v,V_v &= \mathrm{QKV}_v(\mathrm{LN}_v(x_v)), \\
Q_t,K_t,V_t &= \mathrm{QKV}_t(\mathrm{LN}_t(x_t)), \\
[x_v';x_t'] &= \mathrm{SelfAttn}([Q_v;Q_t], [K_v;K_t], [V_v;V_t]).
\end{aligned}
\]

After these initial joint-attention blocks, tokens are concatenated and processed through shared-parameter single-stream blocks for computational efficiency. To prevent semantic drift, each block incorporates reinforced text cross-attention:

\[
x_v'' = x_v' + \mathrm{CrossAttn}(Q=x_v', K=x_t, V=x_t).
\]

Bidirectional cross-attention is applied such that video streams attend to updated audio features and vice versa, facilitating tight temporal and semantic alignment:

\[
\begin{aligned}
a_i' &= a_i + \mathrm{CrossAttn}(Q=a_i, K=v_i, V=v_i),\\
v_i'' &= v_i' + \mathrm{CrossAttn}(Q=v_i', K=a_i', V=a_i').
\end{aligned}
\]

For temporal synchronization, both video and audio latents use 3D Rotary Positional Embeddings (RoPE), scaling the audio frequency such that temporally corresponding tokens share a unified attention grid. The shared MMLM text encoder processes multi-modal prompts—consisting of text, image, video, mask, and audio samples—whose embeddings are injected into self- and cross-attention layers throughout both branches, ensuring coherent conditioning across all modalities.

## 2. Diffusion and Flow-Matching Sequence Modeling

SkyReels-V4 frames video and audio generation as diffusion-based sequence forecasting, adopting a continuous-time flow-matching objective. The forward diffusion process applies Gaussian noise, parameterized as follows for latent $z_0$ and noise $\varepsilon \sim \mathcal N(0, I)$ at diffusion time $t \in [0, 1]$:

\[
z_t = t\,\varepsilon + (1 - t)\,z_0,
\]

with $q(z_t \mid z_0) = \mathcal N(z_t; \sqrt{\alpha_t} z_0, (1-\alpha_t) I), \quad \alpha_t = 1 - t.$

The model predicts the velocity field $v_\theta(t, z_t)$, approximating $z_0 - \varepsilon$, optimizing the mean-squared error loss:

\[
\mathcal{L}_{flow} = \mathbb{E}_{t, z_0, \varepsilon}[\| v_\theta(t, z_t, c) - (z_0 - \varepsilon) \|^2].
\]

This formulation is applied separately to each modality but with explicit cross-modal conditioning:

\[
\mathcal{L} = \mathbb{E}[\|v_\theta^v(t, z_v^t, z_a^t, c) - (z_v^0 - \varepsilon_v)\|^2 + \|v_\theta^a(t, z_a^t, z_v^t, c) - (z_a^0 - \varepsilon_a)\|^2].
\]

Inference proceeds via numerical integration (Euler solver) to denoise and recover the original latent $z_0$.

## 3. Unified Channel-Concatenation Inpainting and Editing

SkyReels-V4 unifies a broad range of video generation, inpainting, extension, and editing tasks using a channel-concatenation approach to masked diffusion. The input tensor is defined as:

\[
Z_{in} = \mathrm{Concat}(V, I, M),
\]

where $V \in \mathbb{R}^{T \times H \times W \times C}$ is the noisy video latent at timestep $t$, $I$ contains VAE-encoded conditional frames (with zeroed unconstrained regions), and $M \in \mathbb{R}^{T \times H \times W \times 1}$ is a binary spatial-temporal mask (1 to preserve, 0 to generate). Specific task configurations are enumerated below:

| Task                    | Mask $M$ Specification                     | Description                              |
|-------------------------|--------------------------------------------|------------------------------------------|
| Text-to-Video (T2V)     | $M \equiv 0$                               | Entire generation                        |
| Image-to-Video (I2V)    | $M_{t=0}=1$, others $=0$                   | Condition on 1st frame                   |
| Video Extension         | $M_{t < k}=1$                              | Extend with last $k$ frames as context   |
| Frame Interpolation     | $M_{0,T-1}=1$                              | Interpolate between given endpoints      |
| Spatiotemporal Editing  | Arbitrary $M$                              | Free-form edit via mask specification    |

This method allows flexible, unified treatment of inpainting-style operations, including the injection of vision-referenced inpainting and editing using multi-modal prompts.

## 4. Efficient High-Resolution Generation Pipeline

To address the computational challenges of 1080p, 32 FPS, and 15 s video-audio generation, SkyReels-V4 implements a staged pipeline:

1. **Joint Low-Resolution and Keyframe Prediction**: The model generates a full-sequence low-resolution video (e.g., 270p/480 frames) and predicts a sparse set of high-resolution keyframes (e.g., every 8th frame at 1080p).
2. **Refiner for Super-Resolution and Interpolation**: A dedicated Refiner module assembles the final video by upsampling low-res latents to a high-res grid, insertion of predicted high-res keyframes, and concatenation with fresh noise for a second DiT pass. The Refiner simultaneously handles video super-resolution (VSR) and frame interpolation via a unified Transformer.
3. **Video Sparse Attention (VSA)**: Each Refiner block adopts a two-stage attention mechanism—pooled attention to identify top-K spatio-temporal cubes followed by dense attention within these cubes—reducing computational cost by approximately 3× with minimal quality degradation.
4. **Training Loss**: The Refiner is trained end-to-end with the same flow-matching loss, extended to jointly optimize high-resolution frames and keyframe interpolation objectives.

This pipeline enables high-fidelity, long-duration outputs with strong temporal consistency across shots and synchronized audio, supporting multi-shot, cinema-level generation [2602.21818].

## 5. Training Regimen and Hyperparameters

SkyReels-V4 is trained through a multi-phase schedule leveraging massive scale across both visual and audio data:

- **Video Pretraining**: Six stages over 11 epochs, starting from T2I at 256px on 3B images (3 epochs), progressing to T2V on 1B images + 400M videos, then expanding to I2V, V2V, and Edit (5% each), mixing resolutions up to 1080px, and finally incorporating image and video references in prompts (~20%).
- **Audio Pretraining**: Training from scratch on hundreds of thousands of hours of speech, music, and SFX up to 15 s, for 3 epochs.
- **Joint Video-Audio Training**: Includes T2V, T2A, and T2AV tasks, using a mixture of video and audio data at higher resolutions and durations (5–15 s).
- **Supervised Fine-Tuning**: Conducted on 5M multi-modal conditioned videos (3 epochs), with a final stage on 1M curated high-quality videos.

Optimization employs AdamW with 0.05 weight decay, learning rate peaking at $2 \times 10^{-5}$ with linear decay, and batch sizes on the order of 2048 per GPU, using 256 GPUs. Diffusion timesteps are sampled as continuous $t \in [0, 1]$, uniformly.

## 6. Multi-Modal Instruction Following and In-Context Applications

SkyReels-V4 accepts and interprets highly structured, multi-modal prompts encompassing text, images, videos, audio, and masks. The MMLM-based text encoder extracts multi-modal embeddings, which are directly integrated throughout both the video and audio synthesis branches. Illustrative prompt types include:

- **Text + Images + Audio**: Prompts resolving to synchronized, semantically rich video and audio (e.g., character identity, lip sync, structured dialogue, camera motion, background matching, and musical overlays).
- **Image-to-Video + Mask-Based Editing**: Enables targeted removal or replacement (e.g., object substitution across all frames), with the audio branch regenarating congruent ambient effects.
- **Motion Transfer**: Supports animation of static images using motion extracted from reference videos, aligning style and pose, and transferring environment and SFX.
- **First-Frame + Artistic Effect**: Applies region-specific transformations over a video sequence, importing effects or styles from reference media via in-context attention.

All such tasks are mediated by the unified channel-concatenation and RoPE-based time-offset mechanisms, supporting extensive editing, extension, and compositional diversity. The architecture integrates cross-modal attention at all abstraction levels, supporting fine-grained control and cross-stream synchronization.

---

SkyReels-V4 advances foundation models for video and audio by providing a robust, extensible framework for unified generation, inpainting, and editing, supporting instructions and contextual guidance across an expansive range of modalities while sustaining computational tractability at cinematic output resolutions [2602.21818].

Source: https://www.emergentmind.com/topics/skyreels-v4