LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

Abstract: Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.

• The paper introduces a novel retrieval-augmented framework that leverages a searchable latent memory to correct long-range temporal coherence errors.
• It integrates local context with retrieved historical latents via compact query embeddings to minimize error accumulation and identity drift.
• Experiments demonstrate significant improvements in subject consistency, motion smoothness, and imaging quality across various autoregressive backbones.

LongLive-RAG: Retrieval-Augmented Autoregressive Video Generation

Motivation and Challenges in Long Video Generation

Autoregressive (AR) video diffusion models have advanced the quality of short clip synthesis, but scaling to long videos presents persistent challenges: error accumulation and identity drift as generation horizons extend. Sliding-window attention, the dominant paradigm for maintaining manageable computation, inherently limits global temporal coherence. As a result, any error or appearance drift in the sliding window history is recursively propagated, leading to deteriorated subject and background consistency over time.

Previous mitigation strategies involve mechanisms such as fixed attention-sinks, positional extrapolation ($\infty$-RoPE), or compressed-history tokens (Deep Forcing). However, these approaches present limitations—anchors may not align with evolving content, positional extensions cannot correct accumulated errors once drift propagates, and token compression risks discarding essential visual cues necessary for scene continuity.

LongLive-RAG Framework

LongLive-RAG formulates the video generation process as a retrieval-augmented generation (RAG) problem, introducing a searchable, content-addressable memory constructed over self-generated latents. Instead of a fixed or heuristically compressed summary, LongLive-RAG enables the AR generator to query its own latent history, retrieving and incorporating globally relevant context for each new block of generation.

The key architectural modifications are minimal but impactful:

• At each block $t$, a compact query embedding, produced by a trained latent encoder, is used to retrieve top-$K$ relevant historical latents from a growing memory pool.
• Retrieved non-local latents are concatenated with local sliding-window context as attention inputs for the generator, enabling both local continuity and global consistency recovery without modification of the denoising backbone.
• The retrieval mechanism is computationally efficient, adding negligible inference overhead dominated by latent encoding.

  Figure 1: LongLive-RAG enables the generator to retrieve visually and semantically relevant segments from its generated history, stabilizing long-range temporal consistency.

  

  Figure 2: Architecture overview: the generator produces latents, which are encoded for retrieval; retrieved context is used in attention, providing continual implicit correction.

Embedding Learning for Discriminative Retrieval

Simply compressing latents via reconstruction is insufficient: temporally adjacent states are overly similar, causing retrieval to focus on redundant frames already present in the local window. To address this, LongLive-RAG introduces the Window Temporal Delta Loss, which penalizes excessive similarity for temporally nearby embeddings, coupled with a smoothing term to maintain stability in the embedding trajectory.

This dual-objective:

1. Encourages embeddings to recognize and separate visually redundant local states.
2. Ensures smoothness such that context selection remains stable and meaningful across long sequences.

   Figure 3: Embedding-space analysis—temporal delta loss reduces over-similarity within local windows, while smoothing stabilizes retrieval embeddings along the generation trajectory.

Experimental Evaluation

Benchmarks and Setup

LongLive-RAG is evaluated across three strong AR backbones (Causal-Forcing, Self-Forcing, LongLive) and variable generation lengths (30s, 60s, 120s). Baselines include base models, positional extrapolation ($\infty$-RoPE), and compressed-history tokens (Deep Forcing). VBench-Long metrics provide comprehensive evaluation, including subject/background consistency, motion smoothness, and imaging quality.

Numerical Results

• Across all backbone-duration pairs, LongLive-RAG achieves the best average VBench-Long ranks.
• Improvements are especially pronounced for subject/background consistency and imaging quality, with retrieval-based context integration reducing color shift, duplication, and background artifacts seen in baseline models.

Figure 4: Qualitative comparison—LongLive-RAG maintains temporal and visual coherence where baselines exhibit artifacts such as subject duplication and abrupt color shifts.

• Auxiliary VLM scoring confirms improved perceptual exposure quality, matching VBench trends.

Retrieval Budget and Ablation Studies

• Experiments varying $K$ (retrieval slots) demonstrate that a balanced allocation (e.g., $K=6$ of 12 total context slots) best trades off local continuity and global reference.
• Learned embeddings significantly outperform random retrieval, average-pooled descriptors, and reconstruction-only encodings.

  Figure 5: Retrieval-budget ablation—an optimal split between local and retrieved context maximizes long-range stability and continuity.

  

  Figure 6: Learned LongLive-RAG embeddings retrieve globally relevant segments, outperforming non-learned strategies in context coverage and utility.

Analysis and Theoretical Implications

LongLive-RAG shifts the context selection paradigm from position-based or summary-based to content-addressed retrieval, dynamically adapting to drift and unexpected appearance changes. This reframes the problem as contextual relevance estimation: the generator is no longer chained to error-prone local history but actively references representative past states.

This approach holds several theoretical implications:

• Retrieval over self-generated latent trajectories echoes mechanisms in retrieval-augmented LLMs, but adapted to high-dimensional generative contexts.
• Embedding learning aligns the retrieval space with the actual needs of the AR generator, bridging the gap between memory efficiency and denoising efficacy.
• The method is architecture-agnostic: it operates with frozen generators and generalizes across backbones that share decoding latent space.

Practical Applications and Future Directions

LongLive-RAG makes practical, scalable long video generation feasible for use cases requiring high temporal consistency (film-making, simulation, interactive world modeling). The minimal computational cost and the compatibility with diverse AR backbones facilitate its deployment in real-world systems.

Potential future research directions include:

• Joint, online training of retrieval encoders and backbones for even tighter integration.
• Extension to multimodal or conditional retrieval (e.g., using coarse scene or semantic layout indexes).
• Exploration of hybrid memory architectures combining learned retrieval with structured priors (e.g., geometric or semantic alignment in world models).

Conclusion

LongLive-RAG presents a robust, general framework for retrieval-augmented long video generation, overcoming core limitations of sliding-window and summary-based context selection. By content-addressably exposing the generator to its own useful historical states, it significantly enhances subject and scene consistency in long AR rollouts while incurring low computational overhead. The methodology provides a foundation for future studies in memory-augmented generative architectures in vision and beyond.