---
title: 'LongCat-Video: Unified Long Video Generation'
url: https://www.emergentmind.com/topics/longcat-video
type: topic
---

# LongCat-Video: Unified Long Video Generation

LongCat-Video is a large-scale, unified video generation foundation model optimized for efficient, high-fidelity, temporally coherent long video synthesis and continuation. With 13.6 billion parameters and a backbone based on the Diffusion Transformer (DiT) framework, LongCat-Video delivers strong empirical performance in text-to-video, image-to-video, and video-continuation tasks. Its architecture and training methodology prioritize both extensibility across video generation paradigms and computational efficiency for minutes-long, high-resolution video output [2510.22200].

## 1. Architecture and Core Components

### 1.1 Diffusion Transformer Backbone

A pretrained WAN2.1 VAE first encodes RGB videos of shape $(T \times H \times W \times 3)$ into compressed latents $z_0$ of shape $\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$, achieving $4\times$ temporal and $16 \times 16$ spatial downsampling. A subsequent patchify step within the DiT further compresses by $1 \times 2 \times 2$, for an effective $4 \times 16 \times 16$ compression.

The DiT core comprises 48 layers (hidden size 4096, FFN size 16384, 32 attention heads, AdaLN-Zero dimension 512) with:

- 3D self-attention: RMSNorm/QKNorm normalization, 3D-RoPE positional encodings.
- Cross-attention: Text-to-video conditioning when prompt present.
- SwiGLU feed-forward blocks.
- AdaLN-Zero conditional modulation for timestep and context.

### 1.2 Unified Multi-Task Conditioning

A single DiT handles Text-to-Video, Image-to-Video, and Video-Continuation by temporally concatenating conditioning frames $X_{cond} \in \mathbb{R}^{B \times N_{cond} \times H \times W \times C}$ with noisy frames $X_{noisy} \in \mathbb{R}^{B \times N_{noisy} \times H \times W \times C}$. During training, the model learns to denoise $N_{noisy}$ frames while preserving input conditioning, supporting efficient task switching.

Block-causal attention with KV (Key/Value) caching enables self-attention within $X_{cond}$ alone, with $X_{noisy}$ tokens attending to both $K_{cond}$ and $K_{noisy}$—conditioning tokens are cached to minimize per-step computation.

### 1.3 Block Sparse Attention (BSA)

To scale attention computationally for long videos at high spatial resolution, Block Sparse Attention partitions the latent sequence of length $S = T \cdot H \cdot W$ into non-overlapping 3D blocks of $(t \times h \times w)$ voxels. Query and key tensors are average-pooled within each block, and the attention mask $M$ retains only top-$r$ key blocks per query block. The result is a reduction in computational complexity from $O(S^2)$ to $O(rS)$ for each attention layer.

Typical BSA pipeline:

1. Pool queries and keys within each block.
2. Compute block-level similarity scores.
3. Select top-$r$ key blocks per query.
4. Mask full attention matrix and apply softmax over unmasked entries only.

### 1.4 Coarse-to-Fine Generation

Generation proceeds in two temporal-spatial passes:

- **Stage 1:** Generate at 480p/15fps with 16 steps (coarse pass).
- **Stage 2:** Upsample to 720p/30fps, then add moderate noise ($t_{thresh} = 0.5$), followed by refinement using a LoRA-fine-tuned expert network with only five denoising steps.

This two-stage pipeline enables rapid synthesis at lower resolutions, with targeted refinement to address fine details and coherence at target output quality.

## 2. Mathematical Formalization

### 2.1 Flow Matching Diffusion

Generation follows a rectified flow approach, distinct from classical DDPMs:

\[
x_t = (1-t) x_0 + t \epsilon, \quad \epsilon \sim \mathcal N(0, I),\quad t \in [0,1]
\]
\[
v_t^* = x_0 - \epsilon
\]

The model $v_\theta$ is trained to predict velocity on noisy samples $(x_t, c, t)$ using the loss:

\[
L = \mathbb{E}_{x_0, \epsilon, t, c} \left\| v_{\mathrm{pred}}(x_t, c, t; \theta) - v_t^* \right\|^2
\]

Sampling equates to discretized reverse SDE steps:

\[
x_{t-\Delta t} = x_t + \mathrm{drift}_\theta(x_t, t, c) \Delta t + \sigma_t \sqrt{\Delta t} \cdot \epsilon
\]
\[
\sigma_t = \sqrt{\frac{t}{1-t}},\quad \mathrm{drift}_\theta = v_\theta + \frac{\sigma_t^2}{2t}[x_t + (1-t)v_\theta]
\]

### 2.2 Coarse-to-Fine Refinement

Refinement expert loss is:

\[
\mathrm{MSE}\left(v_\theta, \frac{x_0-x_{\mathrm{thresh}}}{t_{\mathrm{thresh}}}\right)
\]

where $x_{\mathrm{thresh}}$ is the noisy upsampled latent, and the goal is rapid denoising over a small, moderate-noise range.

## 3. Training Paradigms and Inference

### 3.1 Data and Curriculum

Pretraining is performed on a deduplicated, captioned, and style-tagged large-scale video pool. Progressive curriculum learning involves successive stages focusing on increasing resolution, frame length, and task diversity:

| Stage           | Data Domain      | Steps      | LR    |
|-----------------|------------------|------------|--------|
| 1               | Txt2Img @256p    | 285K       | 1e-4   |
| 2               | Txt2Img+T2V      | 140K       | 1e-4   |
| 3               | All tasks @256p  | 164K       | 5e-5   |
| 4               | All @480p        | 36K        | 5e-5   |
| 5               | 480p+720p        | 53K        | 2e-5   |

Supervised fine-tuning is conducted on a filtered, quality-controlled high-res set.

### 3.2 Multi-Reward RLHF

LongCat-Video employs multi-reward Reinforcement Learning from Human Feedback (RLHF) via Generalized Reward Policy Optimization (GRPO). Four normalized reward signals are used:

- HPSv3-general (visual quality)
- HPSv3-percentile (text alignment on salient frames)
- MQ (VideoAlign on grayscale)
- TA (VideoAlign on color)

Combined policy loss at time $t$:

\[
\mathcal{L}_{\mathrm{policy, multi}}(\theta) = r_t(\theta)\sum_{k=1}^4 w_k\hat{A}_{k,t}
\]
where $\hat{A}_{k,t} = (R_k-\mu_k)/\sigma_{\max,k}$.

A KL-divergence penalty to the reference policy is enforced. LoRA adapters with dimension 128 are employed to support parameter-efficient policy adaptation.

### 3.3 Inference Optimization

The model leverages:

- Classifier-Free Guidance (CFG)
- Consistency Model (CM) distillation

This combination enables the base model to achieve 16-step sampling (down from 50). The full 480p to 720p, 93-frame sequence takes $\sim$116 s on a single H800 GPU, providing a 12.3$\times$ speedup compared to native high-res, long-step sampling.

## 4. Quantitative and Qualitative Results

### 4.1 Text-to-Video and Image-to-Video Performance

Evaluated on internal Mean Opinion Scores (MOS), LongCat-Video achieves:

| Task              | Text Alignment | Visual Quality | Motion Quality | Overall |
|-------------------|---------------|----------------|---------------|---------|
| Text-to-Video     | 3.57          | 3.76           | 3.65          | 3.61    |
| Image-to-Video    | 4.04          | 3.27 (max)     | 3.59          | 3.17    |

On public VBench 2.0, LongCat-Video attains 62.1% overall, outperforming all open-source models for Commonsense (70.9%).

### 4.2 Generation Characteristics

- Natural long-video continuations are achieved without color drift or temporal artifacts.
- Accurate instruction following in camera moves, actions, and style changes.
- Textural and fine detail improvements via coarse-to-fine LoRA expert refinement.

## 5. Significance and Relation to World Models

LongCat-Video is the first open, 13.6B-parameter unified video foundation model to be natively pretrained for extended video continuation. By tightly integrating flow-matching diffusion, task generalization, block-sparse attention, and efficient hierarchical generation, it makes practical the generation of minutes-long, 720p 30fps content within minutes on a single GPU.

This capacity for temporally robust, high-fidelity video generation is a foundational enabling technology for large-scale "world models": architectures capable of modeling, simulating, and planning in rich, temporally extended spatiotemporal environments. The open availability of code and weights accelerates reproducibility and research in robust long-horizon video generation [2510.22200].

Source: https://www.emergentmind.com/topics/longcat-video