LongLive-2.0: Efficient Parallel Video Generation

• LongLive-2.0 is a parallel infrastructure for long video generation, leveraging NVFP4 low-precision arithmetic and optimized hardware utilization.
• It implements a sequence-parallel autoregressive (AR) training method, 'Balanced SP', ensuring efficient memory use and high-throughput performance on advanced GPUs.
• The system integrates LoRA adapters and direct AR diffusion tuning to reduce denoising steps for real-time inference while significantly lowering training latency and memory requirements.

LongLive-2.0 is a parallel infrastructure designed explicitly for long video generation, leveraging the NVFP4 4-bit floating-point format throughout the end-to-end workflow for both training and inference. The system integrates a sequence-parallel, autoregressive (AR) training method referred to as "Balanced SP" and co-designs the model, data layout, and hardware utilization for efficient, scalable, and high-quality generation of video content. LongLive-2.0 departs from conventional, multi-stage training pipelines by directly tuning a diffusion model into a long, multi-shot AR diffusion model, and further enables real-time inference at high throughput with modular optimizations tailored for the latest Blackwell (GB200) and other advanced GPU architectures (Chen et al., 18 May 2026).

1. NVFP4 Numeric Format and Hardware Utilization

LongLive-2.0 employs NVFP4—a 4-bit floating-point format with a per-block FP8 scale and a global FP32 scale—for all tensors in both training and inference. Each element is represented as

$$\hat X = X^{\mathrm{FP4}} \times \alpha^{\mathrm{FP8}} \times \alpha^{\mathrm{FP32}},$$

where $X^{\mathrm{FP4}}$ is a 4-bit E2M1 mantissa, $\alpha^{\mathrm{FP8}}$ is an FP8 E4M3 scale per 16-element block, and $\alpha^{\mathrm{FP32}}$ is a global scale factor. This structure minimizes memory traffic, especially in generalized matrix multiplications (GEMMs), theoretically reducing traffic by $4\times$ compared to BF16 with practical speedup measured at $\approx 1.8\times$ on GB200 hardware.

For working memory, if $M_{\mathrm{BF16}}$ is the BF16 memory requirement, then $$M_{\mathrm{NV4}} \approx \tfrac{1}{2} M_{\mathrm{BF16}}$$, discounting scale overhead. Key-value (KV) cache quantization into NVFP4 with $T_c \times H \times d$ block size reduces storage from $4 T_c H d$ bytes (BF16) to approximately $X^{\mathrm{FP4}}$0—a $X^{\mathrm{FP4}}$1 compression.

2. Sequence-Parallel Autoregressive Training: Balanced SP

The "Balanced SP" method performs autoregressive diffusion training at scale and in parallel. Training proceeds as follows:

• Raw videos, partitioned into $X^{\mathrm{FP4}}$2 consecutive chunks $X^{\mathrm{FP4}}$3, are VAE-encoded to obtain latent frames $X^{\mathrm{FP4}}$4.
• GPUs are grouped into SP groups of size $X^{\mathrm{FP4}}$5; each GPU rank $X^{\mathrm{FP4}}$6 owns one chunk plus a left "halo" for the VAE receptive field. The VAE encoding cost per rank thus becomes $X^{\mathrm{FP4}}$7 instead of $X^{\mathrm{FP4}}$8 for the entire sequence.
• On each rank, paired streams $X^{\mathrm{FP4}}$9 are formed, where

$\alpha^{\mathrm{FP8}}$0

Here, $\alpha^{\mathrm{FP8}}$1 contains ground-truth latent chunks, and $\alpha^{\mathrm{FP8}}$2 applies a local DDPM noise schedule.

A distinctive aspect is the use of an SP-native teacher-forcing mask to facilitate interleaving clean and noisy tokens across all ranks after an all-to-all swap of sequence and head dimensions. Masking is determined using local indices, which avoids expensive reshuffling.

The Balanced SP algorithm (briefly summarized):

1. Each rank loads its raw chunk plus halo, VAE-encodes, and applies noise.
2. All-to-all swaps gather Q/K/V in NVFP4 format.
3. Local construction of the natural-order teacher-forcing mask enables a single fused attention pass.
4. Loss computation and all-reduce occur uniformly, followed by in-place NVFP4 model updates via a random Hadamard transform on both gradients and block scales.

3. Direct AR Diffusion Tuning and LoRA Distillation

The direct fine-tuning approach initializes from a bidirectional video DiT and trains on long, multi-shot videos with an AR objective:

$\alpha^{\mathrm{FP8}}$3

where $\alpha^{\mathrm{FP8}}$4 is the noisy latent of chunk $\alpha^{\mathrm{FP8}}$5 at timestep $\alpha^{\mathrm{FP8}}$6, and $\alpha^{\mathrm{FP8}}$7 is the denoiser conditioned on clean history.

This single-stage pipeline eliminates the need for ODE initialization and explicit distribution matching distillation (DMD) stages used in prior work. Few-step DMD is realized by training lightweight LoRA adapters on a frozen NVFP4-quantized backbone:

$\alpha^{\mathrm{FP8}}$8

with $\alpha^{\mathrm{FP8}}$9, and $\alpha^{\mathrm{FP32}}$0.

This LoRA conversion allows for reduction to as few as 2–4 denoising steps for real-time inference, all within a unified distillation stage.

4. Inference Pipeline and Hardware Optimization

Inference optimizations are adapted to both Blackwell GPUs (GB200) and previous-generation architectures:

• On Blackwell GB200, full W4A4 NVFP4 inference is conducted for all DiT computations, supporting up to $\alpha^{\mathrm{FP32}}$1 faster GEMMs and $\alpha^{\mathrm{FP32}}$2 reduction in end-to-end latency ($\alpha^{\mathrm{FP32}}$3 ms $\alpha^{\mathrm{FP32}}$4 $\alpha^{\mathrm{FP32}}$5 ms per frame). KV-cache quantization further delivers a $\alpha^{\mathrm{FP32}}$6 storage reduction.
• Asynchronous streaming VAE decoding dedicates one GPU for chunked 3D VAE decodes, with CPU offload to overlap VAE and DiT times. End-to-end latency becomes $\alpha^{\mathrm{FP32}}$7 instead of $\alpha^{\mathrm{FP32}}$8 for a video with $\alpha^{\mathrm{FP32}}$9 chunks.
• For non-Blackwell GPUs (e.g., H100, A100), SP-inference is deployed: each rank processes $4\times$0 tokens, using all-to-all NVFP4 communication to exchange Q, K, V tensors. The communication payload and memory are both reduced by $4\times$1 via cache quantization.

5. Empirical Evaluation and Benchmarking

LongLive-2.0 achieves significant acceleration and memory savings across training and inference scenarios, as summarized in the following tables.

AR Training Latency (Seconds)

Config	16s	32s	64s
BF16 w/o SP	75.3	202.7	OOM
BF16 + SP	52.2	162.7	1372.9
BF16 + Balanced SP	45.8	136.8	1196.5
NVFP4 + Balanced SP	40.1	119.3	639.5

DMD LoRA Distillation Memory (Peak, GB)

	Peak GPU Mem
BF16×all	70.5
NV4 Gen	63.3
+LoRA	57.2
All NV4+LoRA	49.0

Inference Throughput (FPS) and Memory (GB)

Setting	FPS	16s (lat., mem.)	32s	64s
BF16 baseline	24.8	26.6s, 36.4GB	53.2s,36.4GB	112.9s,36.4GB
+ NVFP4	32.0	22.9s, 29.7GB	46.6s,29.7GB	96.0s,29.7GB
+ KV-cache compression	29.7	23.8s, 19.4GB	48.9s,19.4GB	99.5s,19.4GB
+ Async VAE	29.7	15.9s, 19.4GB	29.1s,19.4GB	57.6s,19.4GB
3-step	35.2	12.7s, 19.4GB	23.2s,19.4GB	46.0s,19.4GB
2-step	45.7	11.2s, 19.4GB	19.2s,19.4GB	36.3s,19.4GB

VBench and VBench-Long Results

Model	Precision	Steps	Params	Res	FPS	Total Score
LongLive-2.0 (BF16)	BF16	4	5B	1280×720	24.8	85.06
LongLive-2.0 (NVFP4, 4)	NVFP4	4	5B	1280×720	29.7	84.51
LongLive-2.0 (NVFP4, 2)	NVFP4	2	5B	1280×720	45.7	83.14

Method	Subj.C.	Back.C.	Smooth.	Dynamic	Aesthetic	Avg. Rank
LongLive-2.0 (BF16)	97.13	95.89	98.61	44.56	58.17	3.67
LongLive-2.0 (NV4)	97.62	96.97	98.94	45.88	53.72	3.83

LongLive-2.0 demonstrates up to $4\times$2 training speedup and $4\times$3 inference latency reduction over BF16 baselines, real-time 720p generation at 45.7 FPS (2 denoising steps), substantial GPU memory savings, and maintains state-of-the-art quality on established video generation benchmarks.

6. Impact and Distinguishing Features

Key innovations of LongLive-2.0 include:

• First use of NVFP4 low-precision format for both training and inference in long video generation tasks.
• Co-design of the computation and dataflow pipeline for optimal efficiency in teacher-forced sequence-parallel AR training.
• Elimination of multi-stage DMD/ODE preprocessing; all training occurs in a single AR objective-driven stage.
• Modular LoRA adapters enabling rapid distillation to few-step real-time inference.
• Hardware-adaptive inference optimizations, such as W4A4 GEMMs and asynchronous VAE streaming, generalizing performance across diverse GPU classes.

A plausible implication is that, as hardware platforms continue to evolve towards lower-precision operations and scale-out capability, the architectural template pioneered by LongLive-2.0 will become a foundational reference for future long-horizon generative modeling pipelines.

(Chen et al., 18 May 2026)