Papers
Topics
Authors
Recent
Search
2000 character limit reached

LongLive-2.0: Efficient Parallel Video Generation

Updated 20 May 2026
  • LongLive-2.0 is a parallel infrastructure for long video generation, leveraging NVFP4 low-precision arithmetic and optimized hardware utilization.
  • It implements a sequence-parallel autoregressive (AR) training method, 'Balanced SP', ensuring efficient memory use and high-throughput performance on advanced GPUs.
  • The system integrates LoRA adapters and direct AR diffusion tuning to reduce denoising steps for real-time inference while significantly lowering training latency and memory requirements.

LongLive-2.0 is a parallel infrastructure designed explicitly for long video generation, leveraging the NVFP4 4-bit floating-point format throughout the end-to-end workflow for both training and inference. The system integrates a sequence-parallel, autoregressive (AR) training method referred to as "Balanced SP" and co-designs the model, data layout, and hardware utilization for efficient, scalable, and high-quality generation of video content. LongLive-2.0 departs from conventional, multi-stage training pipelines by directly tuning a diffusion model into a long, multi-shot AR diffusion model, and further enables real-time inference at high throughput with modular optimizations tailored for the latest Blackwell (GB200) and other advanced GPU architectures (Chen et al., 18 May 2026).

1. NVFP4 Numeric Format and Hardware Utilization

LongLive-2.0 employs NVFP4—a 4-bit floating-point format with a per-block FP8 scale and a global FP32 scale—for all tensors in both training and inference. Each element is represented as

X^=XFP4×αFP8×αFP32,\hat X = X^{\mathrm{FP4}} \times \alpha^{\mathrm{FP8}} \times \alpha^{\mathrm{FP32}},

where XFP4X^{\mathrm{FP4}} is a 4-bit E2M1 mantissa, αFP8\alpha^{\mathrm{FP8}} is an FP8 E4M3 scale per 16-element block, and αFP32\alpha^{\mathrm{FP32}} is a global scale factor. This structure minimizes memory traffic, especially in generalized matrix multiplications (GEMMs), theoretically reducing traffic by 4×4\times compared to BF16 with practical speedup measured at ≈1.8×\approx 1.8\times on GB200 hardware.

For working memory, if MBF16M_{\mathrm{BF16}} is the BF16 memory requirement, then MNV4≈12MBF16M_{\mathrm{NV4}} \approx \tfrac{1}{2} M_{\mathrm{BF16}}, discounting scale overhead. Key-value (KV) cache quantization into NVFP4 with Tc×H×dT_c \times H \times d block size reduces storage from 4TcHd4 T_c H d bytes (BF16) to approximately (9/8)TcHd(9/8) T_c H d—a ≈3.6×\approx 3.6\times compression.

2. Sequence-Parallel Autoregressive Training: Balanced SP

The "Balanced SP" method performs autoregressive diffusion training at scale and in parallel. Training proceeds as follows:

  • Raw videos, partitioned into NN consecutive chunks {Xi}\{\mathbf X_i\}, are VAE-encoded to obtain latent frames {Zi}\{\mathbf Z_i\}.
  • GPUs are grouped into SP groups of size PP; each GPU rank pp owns one chunk plus a left "halo" for the VAE receptive field. The VAE encoding cost per rank thus becomes O(F/P+h)O(F/P + h) instead of O(F)O(F) for the entire sequence.
  • On each rank, paired streams z(p)\mathbf z^{(p)} are formed, where

z(p)=[ zclean(p),  znoisy(p)]∈RLP×H×d\mathbf z^{(p)} = [\,\mathbf z_{\text{clean}}^{(p)},\; \mathbf z_{\text{noisy}}^{(p)}] \in \mathbb R^{\tfrac{L}{P} \times H \times d}

Here, zclean(p)\mathbf z_{\text{clean}}^{(p)} contains ground-truth latent chunks, and znoisy(p)\mathbf z_{\text{noisy}}^{(p)} applies a local DDPM noise schedule.

A distinctive aspect is the use of an SP-native teacher-forcing mask to facilitate interleaving clean and noisy tokens across all ranks after an all-to-all swap of sequence and head dimensions. Masking is determined using local indices, which avoids expensive reshuffling.

The Balanced SP algorithm (briefly summarized):

  1. Each rank loads its raw chunk plus halo, VAE-encodes, and applies noise.
  2. All-to-all swaps gather Q/K/V in NVFP4 format.
  3. Local construction of the natural-order teacher-forcing mask enables a single fused attention pass.
  4. Loss computation and all-reduce occur uniformly, followed by in-place NVFP4 model updates via a random Hadamard transform on both gradients and block scales.

3. Direct AR Diffusion Tuning and LoRA Distillation

The direct fine-tuning approach initializes from a bidirectional video DiT and trains on long, multi-shot videos with an AR objective:

x^t−1i=αt xti+1−αt2 ϵθ(z<i, xti, t),\hat x^i_{t-1} = \alpha_t\,x^i_t + \sqrt{1-\alpha_t^2}\,\epsilon_\theta(z_{<i},\,x^i_t,\,t),

where xtix^i_t is the noisy latent of chunk ii at timestep tt, and ϵθ\epsilon_\theta is the denoiser conditioned on clean history.

This single-stage pipeline eliminates the need for ODE initialization and explicit distribution matching distillation (DMD) stages used in prior work. Few-step DMD is realized by training lightweight LoRA adapters on a frozen NVFP4-quantized backbone:

W≃Dequant(Qsearch(W0))+αLoRABA,W \simeq \mathrm{Dequant}\bigl(Q_{\text{search}}(W_0)\bigr) + \alpha_{\text{LoRA}} B A,

with A∈Rd×r, B∈Rr×dA \in \mathbb R^{d\times r},\ B \in \mathbb R^{r\times d}, and r≪dr \ll d.

This LoRA conversion allows for reduction to as few as 2–4 denoising steps for real-time inference, all within a unified distillation stage.

4. Inference Pipeline and Hardware Optimization

Inference optimizations are adapted to both Blackwell GPUs (GB200) and previous-generation architectures:

  • On Blackwell GB200, full W4A4 NVFP4 inference is conducted for all DiT computations, supporting up to 4×4\times faster GEMMs and 1.84×1.84\times reduction in end-to-end latency ($40.3$ ms →\rightarrow $21.9$ ms per frame). KV-cache quantization further delivers a 3.6×3.6\times storage reduction.
  • Asynchronous streaming VAE decoding dedicates one GPU for chunked 3D VAE decodes, with CPU offload to overlap VAE and DiT times. End-to-end latency becomes Câ‹…tDiTC \cdot t_{\mathrm{DiT}} instead of Câ‹…(tDiT+tVAE)C \cdot (t_{\mathrm{DiT}} + t_{\mathrm{VAE}}) for a video with CC chunks.
  • For non-Blackwell GPUs (e.g., H100, A100), SP-inference is deployed: each rank processes L/PL/P tokens, using all-to-all NVFP4 communication to exchange Q, K, V tensors. The communication payload and memory are both reduced by 3.6×3.6\times via cache quantization.

5. Empirical Evaluation and Benchmarking

LongLive-2.0 achieves significant acceleration and memory savings across training and inference scenarios, as summarized in the following tables.

AR Training Latency (Seconds)

Config 16s 32s 64s
BF16 w/o SP 75.3 202.7 OOM
BF16 + SP 52.2 162.7 1372.9
BF16 + Balanced SP 45.8 136.8 1196.5
NVFP4 + Balanced SP 40.1 119.3 639.5

DMD LoRA Distillation Memory (Peak, GB)

Peak GPU Mem
BF16×all 70.5
NV4 Gen 63.3
+LoRA 57.2
All NV4+LoRA 49.0

Inference Throughput (FPS) and Memory (GB)

Setting FPS 16s (lat., mem.) 32s 64s
BF16 baseline 24.8 26.6s, 36.4GB 53.2s,36.4GB 112.9s,36.4GB
+ NVFP4 32.0 22.9s, 29.7GB 46.6s,29.7GB 96.0s,29.7GB
+ KV-cache compression 29.7 23.8s, 19.4GB 48.9s,19.4GB 99.5s,19.4GB
+ Async VAE 29.7 15.9s, 19.4GB 29.1s,19.4GB 57.6s,19.4GB
3-step 35.2 12.7s, 19.4GB 23.2s,19.4GB 46.0s,19.4GB
2-step 45.7 11.2s, 19.4GB 19.2s,19.4GB 36.3s,19.4GB

VBench and VBench-Long Results

Model Precision Steps Params Res FPS Total Score
LongLive-2.0 (BF16) BF16 4 5B 1280×720 24.8 85.06
LongLive-2.0 (NVFP4, 4) NVFP4 4 5B 1280×720 29.7 84.51
LongLive-2.0 (NVFP4, 2) NVFP4 2 5B 1280×720 45.7 83.14
Method Subj.C. Back.C. Smooth. Dynamic Aesthetic Avg. Rank
LongLive-2.0 (BF16) 97.13 95.89 98.61 44.56 58.17 3.67
LongLive-2.0 (NV4) 97.62 96.97 98.94 45.88 53.72 3.83

LongLive-2.0 demonstrates up to 2.15×2.15\times training speedup and 1.84×1.84\times inference latency reduction over BF16 baselines, real-time 720p generation at 45.7 FPS (2 denoising steps), substantial GPU memory savings, and maintains state-of-the-art quality on established video generation benchmarks.

6. Impact and Distinguishing Features

Key innovations of LongLive-2.0 include:

  • First use of NVFP4 low-precision format for both training and inference in long video generation tasks.
  • Co-design of the computation and dataflow pipeline for optimal efficiency in teacher-forced sequence-parallel AR training.
  • Elimination of multi-stage DMD/ODE preprocessing; all training occurs in a single AR objective-driven stage.
  • Modular LoRA adapters enabling rapid distillation to few-step real-time inference.
  • Hardware-adaptive inference optimizations, such as W4A4 GEMMs and asynchronous VAE streaming, generalizing performance across diverse GPU classes.

A plausible implication is that, as hardware platforms continue to evolve towards lower-precision operations and scale-out capability, the architectural template pioneered by LongLive-2.0 will become a foundational reference for future long-horizon generative modeling pipelines.

(Chen et al., 18 May 2026)

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to LongLive-2.0.