LongCat-Video: Unified Long Video Generation

• LongCat-Video is a unified video generation model designed for efficient, high-fidelity long video synthesis using a 13.6B-parameter Diffusion Transformer backbone.
• It employs block sparse attention and a coarse-to-fine refinement pipeline to generate temporally coherent high-resolution video content.
• The model supports multi-task conditioning for text-to-video, image-to-video, and video-continuation tasks, achieving strong results in visual and motion quality.

LongCat-Video is a large-scale, unified video generation foundation model optimized for efficient, high-fidelity, temporally coherent long video synthesis and continuation. With 13.6 billion parameters and a backbone based on the Diffusion Transformer (DiT) framework, LongCat-Video delivers strong empirical performance in text-to-video, image-to-video, and video-continuation tasks. Its architecture and training methodology prioritize both extensibility across video generation paradigms and computational efficiency for minutes-long, high-resolution video output (Team et al., 25 Oct 2025).

1. Architecture and Core Components

1.1 Diffusion Transformer Backbone

A pretrained WAN2.1 VAE first encodes RGB videos of shape $(T \times H \times W \times 3)$ into compressed latents $z_0$ of shape $$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$, achieving $4\times$ temporal and $16 \times 16$ spatial downsampling. A subsequent patchify step within the DiT further compresses by $1 \times 2 \times 2$, for an effective $4 \times 16 \times 16$ compression.

The DiT core comprises 48 layers (hidden size 4096, FFN size 16384, 32 attention heads, AdaLN-Zero dimension 512) with:

• 3D self-attention: RMSNorm/QKNorm normalization, 3D-RoPE positional encodings.
• Cross-attention: Text-to-video conditioning when prompt present.
• SwiGLU feed-forward blocks.
• AdaLN-Zero conditional modulation for timestep and context.

1.2 Unified Multi-Task Conditioning

A single DiT handles Text-to-Video, Image-to-Video, and Video-Continuation by temporally concatenating conditioning frames $$X_{cond} \in \mathbb{R}^{B \times N_{cond} \times H \times W \times C}$$ with noisy frames $$X_{noisy} \in \mathbb{R}^{B \times N_{noisy} \times H \times W \times C}$$. During training, the model learns to denoise $N_{noisy}$ frames while preserving input conditioning, supporting efficient task switching.

Block-causal attention with KV (Key/Value) caching enables self-attention within $z_0$0 alone, with $z_0$1 tokens attending to both $z_0$2 and $z_0$3—conditioning tokens are cached to minimize per-step computation.

1.3 Block Sparse Attention (BSA)

To scale attention computationally for long videos at high spatial resolution, Block Sparse Attention partitions the latent sequence of length $z_0$4 into non-overlapping 3D blocks of $z_0$5 voxels. Query and key tensors are average-pooled within each block, and the attention mask $z_0$6 retains only top-$z_0$7 key blocks per query block. The result is a reduction in computational complexity from $z_0$8 to $z_0$9 for each attention layer.

Typical BSA pipeline:

1. Pool queries and keys within each block.
2. Compute block-level similarity scores.
3. Select top-$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$0 key blocks per query.
4. Mask full attention matrix and apply softmax over unmasked entries only.

1.4 Coarse-to-Fine Generation

Generation proceeds in two temporal-spatial passes:

• Stage 1: Generate at 480p/15fps with 16 steps (coarse pass).
• Stage 2: Upsample to 720p/30fps, then add moderate noise ($$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$1), followed by refinement using a LoRA-fine-tuned expert network with only five denoising steps.

This two-stage pipeline enables rapid synthesis at lower resolutions, with targeted refinement to address fine details and coherence at target output quality.

2. Mathematical Formalization

2.1 Flow Matching Diffusion

Generation follows a rectified flow approach, distinct from classical DDPMs:

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$2

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$3

The model $$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$4 is trained to predict velocity on noisy samples $$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$5 using the loss:

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$6

Sampling equates to discretized reverse SDE steps:

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$7

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$8

2.2 Coarse-to-Fine Refinement

Refinement expert loss is:

$$\frac{T}{4} \times \frac{H}{16} \times \frac{W}{16} \times C$$9

where $4\times$0 is the noisy upsampled latent, and the goal is rapid denoising over a small, moderate-noise range.

3. Training Paradigms and Inference

3.1 Data and Curriculum

Pretraining is performed on a deduplicated, captioned, and style-tagged large-scale video pool. Progressive curriculum learning involves successive stages focusing on increasing resolution, frame length, and task diversity:

Stage	Data Domain	Steps	LR
1	Txt2Img @256p	285K	1e-4
2	Txt2Img+T2V	140K	1e-4
3	All tasks @256p	164K	5e-5
4	All @480p	36K	5e-5
5	480p+720p	53K	2e-5

Supervised fine-tuning is conducted on a filtered, quality-controlled high-res set.

3.2 Multi-Reward RLHF

LongCat-Video employs multi-reward Reinforcement Learning from Human Feedback (RLHF) via Generalized Reward Policy Optimization (GRPO). Four normalized reward signals are used:

• HPSv3-general (visual quality)
• HPSv3-percentile (text alignment on salient frames)
• MQ (VideoAlign on grayscale)
• TA (VideoAlign on color)

Combined policy loss at time $4\times$1:

$4\times$2

where $4\times$3.

A KL-divergence penalty to the reference policy is enforced. LoRA adapters with dimension 128 are employed to support parameter-efficient policy adaptation.

3.3 Inference Optimization

The model leverages:

• Classifier-Free Guidance (CFG)
• Consistency Model (CM) distillation

This combination enables the base model to achieve 16-step sampling (down from 50). The full 480p to 720p, 93-frame sequence takes $4\times$4116 s on a single H800 GPU, providing a 12.3$4\times$5 speedup compared to native high-res, long-step sampling.

4. Quantitative and Qualitative Results

4.1 Text-to-Video and Image-to-Video Performance

Evaluated on internal Mean Opinion Scores (MOS), LongCat-Video achieves:

Task	Text Alignment	Visual Quality	Motion Quality	Overall
Text-to-Video	3.57	3.76	3.65	3.61
Image-to-Video	4.04	3.27 (max)	3.59	3.17

On public VBench 2.0, LongCat-Video attains 62.1% overall, outperforming all open-source models for Commonsense (70.9%).

4.2 Generation Characteristics

• Natural long-video continuations are achieved without color drift or temporal artifacts.
• Accurate instruction following in camera moves, actions, and style changes.
• Textural and fine detail improvements via coarse-to-fine LoRA expert refinement.

5. Significance and Relation to World Models

LongCat-Video is the first open, 13.6B-parameter unified video foundation model to be natively pretrained for extended video continuation. By tightly integrating flow-matching diffusion, task generalization, block-sparse attention, and efficient hierarchical generation, it makes practical the generation of minutes-long, 720p 30fps content within minutes on a single GPU.

This capacity for temporally robust, high-fidelity video generation is a foundational enabling technology for large-scale "world models": architectures capable of modeling, simulating, and planning in rich, temporally extended spatiotemporal environments. The open availability of code and weights accelerates reproducibility and research in robust long-horizon video generation (Team et al., 25 Oct 2025).