---
title: 'Open-Sora-Plan: Open-Source Video Generation Suite'
url: https://www.emergentmind.com/topics/open-sora-plan
type: topic
---

# Open-Sora-Plan: Open-Source Video Generation Suite

Open-Sora-Plan is an open-source, large-scale video generation suite that provides an integrated architecture and workflow for producing high-resolution, long-duration video synthesis from multiple conditioning sources. It encompasses novel model components, optimized training/inference strategies, and multi-dimensional data curation, aiming for both high efficiency and strong quantitative/qualitative outcomes. All code, model weights, and documentation are publicly released for academic and applied use [2412.00131].

## 1. Wavelet-Flow Variational Autoencoder (WF-VAE)

The WF-VAE forms the foundational latent-space representation for Open-Sora-Plan. It consists of encoder and decoder modules leveraging 3D convolutional backbones. Compression is realized through cascaded multi-level Haar wavelet transforms, decomposing the input video $\mathbf{V} \in \mathbb{R}^{C\times T\times H\times W}$ into eight subbands per scale:

\[
\mathbf{S}^{(\ell)}_{ijk} = \mathbf{S}^{(\ell-1)} * (f_i \otimes f_j \otimes f_k) \text{ with } h = \tfrac{1}{\sqrt2}[1, 1], g = \tfrac{1}{\sqrt2}[1, -1]
\]

A main energy path ("Flow") injects low-frequency wavelet features into the decoder, supporting symmetry and structural consistency across scales.

The VAE objective follows standard ELBO-based training augmented with:

- $\mathcal{L}_{\ell_1}$ and $\lambda_{LPIPS} \mathcal{L}_{LPIPS}$ for reconstruction.
- $\mathcal{L}_{adv}$ adversarial loss with dynamic reweighting ($\lambda_{adv}$).
- $\lambda_{WL} \mathcal{L}_{WL}$ for wavelet-flow consistency, enforcing feature preservation in decomposed bands.
- KL regularization.

The VAE achieves compression via $4 \times 8 \times 8$ in $(T \times H \times W)$, followed by $1 \times 2 \times 2$ patch embedding. Block-wise inference utilizes causal 3D convolutions and a cache-based tile boundary handling:

\[
T_{cache}(m) = k_t + m\,T_{chunk} - s_t \Bigl\lfloor\frac{m\,T_{chunk}}{s_t} + 1\Bigr\rfloor
\]

This approach eliminates boundary artifacts during streaming inference.

## 2. Joint Image-Video Skiparse Denoiser

The Skiparse Denoiser operates on VAE-compressed latent space, providing efficient and scalable denoising via a sparse attention mechanism:

- Embedding through 3D convolution for video patches (stride $k_t=1$, $k_h=k_w=2$).
- Text encoding via mT5-XXL; embedded prompt projected for cross-attention.
- AdaLN-Zero adapts per timestep with scale/shift/gate injection.
- 3D RoPE (Rotary Positional Embedding) applied by feature partition.

Skiparse Attention alternates between "single skip" and "group skip" spatial patterns, reducing complexity by $\sim 1/k$ without loss of cross-attention. The average attention distance for pattern $k$ is

\[
AD_{avg} \approx 2 - \frac{2}{k} + \frac{1}{k^2}, \qquad (1<k \ll THW)
\]

The denoising loss uses v-prediction with min-SNR reweighting (threshold $\gamma=5.0$):

\[
\mathcal{L}_{\text{vpred}} = \mathbb{E}_{x_0, \epsilon, t} [w(t) \|v_\theta(x_t, t) - v\|^2], \quad w(t) = \min(\mathrm{snr}(t), \gamma).
\]

Training proceeds in stages: initial T2I pretrain (100k steps full 3D; 100k Skiparse), followed by joint image/video pretrain (200k steps), and fine-tuning (30k steps) on filtered video data.

## 3. Condition Controllers

Open-Sora-Plan includes a suite of controllers for diverse conditioning modalities:

- **Text Controller**: Encodes prompts (via mT5-XXL, projected for cross-attn keys/values).
- **Image Condition Controller**: Implements temporal inpainting for image-to-video, transition, and continuation. Noisy latent, mask, and masked video latent are concatenated along the channel dimension for U-Net processing.
- **Structure Condition Controller**: Guides generation by adding representations from edge, depth, or sketch conditions to latent tokens per transformer block: $X_j \leftarrow X_j + F_j$ where $F_j$ is an encoded representation projected to match $X_j$ dimensions.

Structure guidance is trained on Panda70M for 20k steps using a light 3D convolutional encoder and transformer blocks.

## 4. Assistant Strategies for Training and Inference

Several operational strategies optimize scalability, efficiency, and robustness:

- **Min-Max Token Strategy**: Unifies batches of different resolutions/durations by selecting a downsample factor $k$ across aspect-ratio buckets, yielding constant token count per batch.
- **Adaptive Gradient Clipping**: Gradient norms ($gn_i$) are dynamically distributed: abnormal GPUs are zeroed, normal ones are rescaled $(N/M \times g_i)$, with all-reduce performed afterward. EMA tracking prevents isolated loss spikes.
- **Prompt Refiner**: LoRA-adapted LLaMA-3.1-8B is fine-tuned to expand prompts for improved semantic richness and VBench scores. During inference, user prompts are rewritten for optimal conditioning.

## 5. Multi-Dimensional Data Curation Pipeline

Data collection and processing comprise image and video datasets such as SAM (11.1M), Anytext (1.8M), Panda70M (21.2M), VIDAL (2.8M), and internal sources. Filtering is multi-step:

- Slicing to 16s.
- Jump-cut detection via LPIPS (removes 3%).
- Motion filter (mean LPIPS within [0.001, 0.3], retains 89%).
- OCR cropping (EasyOCR removes up to 20% edges).
- Aesthetics (Laion pred $\geq$ 4.75, 49% remain).
- Technical quality (DOVER $\geq$ 0, 44%).
- Final motion double-check ($\sim$42%).
- Captions via LLaVA, InternVL2, Qwen2-VL (images), Qwen2-VL, ShareGPT4Video (videos).

## 6. Quantitative and Qualitative Evaluation

WF-VAE exhibits SOTA or superior performance relative to CV-VAE, OD-VAE, Allegro, and CogVideoX. Reconstruction metrics (33-frame @256²):

- PSNR, LPIPS, SSIM, rFVD all indicate competitive or best-in-class results.
- WF-VAE-S achieves 11.11 vids/s @512²×33, 4.7GB memory.
- Causal Cache preserves lossless tiling (no reconstruction degradation).

Text-to-video: OpenSora v1.2 (1.2B) and OpenSora Plan v1.3 (2.7B) outperform in VBench (Aesthetic=59.00, Action=81.8, Scene=71.00; prompt-refined: Object=84.7, Scene=52.9, MTScore=2.95).

Image-to-video and structure-video workflows are benchmarked via qualitative displays (I2V, transitions, arbitrary structure insertion).

## 7. Implementation and Usage

All Open-Sora-Plan resources and pretrained weights are distributed via [https://github.com/PKU-YuanGroup/Open-Sora-Plan]. The Python interface supports:

```python
from opensora import VideoGenerator
gen = VideoGenerator(model="opensora-plan-v1.3")
out = gen.generate(text="…", seed=42, steps=50, cfg=7.5)
```

Supports text-to-video, image-to-video, inpainting, structure-guided modes, and multilingual prompting (integrated refiner). Mixed-precision inference (fp16) and patch-based tiling are natively supported [2412.00131].

All components, pipelines, and evaluation metrics are detailed explicitly in the primary publication to facilitate direct reproduction and further research.

Source: https://www.emergentmind.com/topics/open-sora-plan