LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

When generating long videos with AI, characters and backgrounds often change or look weird over time. This instability happens because most models predict one small piece of video after another. Eventually, a tiny mistake in one piece compounds. This leads to a total breakdown in visual logic.

Current state-of-the-art approaches use autoregressive (AR) video diffusion. These models generate video by emitting latent blocks (compressed mathematical representations of video) causally, or one after another. To keep computation manageable, these models typically use a sliding-window attention mechanism. This means the model only looks at the most recent few seconds of video to decide what happens next. While efficient, this creates an irreversible trajectory. If the recent window accumulates an error—like a character's hair changing color—the model treats that error as ground truth. It then builds upon it and drifts further away from the original intent.

The Problem

The status quo in long-horizon video generation is essentially a game of telephone played with pixels. Because sliding-window attention discards earlier context to bound memory, the generator loses sight of the "source of truth" established at the start. As shown in, once the local context drifts off the "video manifold" (the mathematical space representing realistic imagery), subsequent blocks are forced to condition on this degraded state.

Existing attempts to fix this involve keeping fixed "anchors" (early frames held constant) or using positional extrapolation to stretch the model's sense of time. However, the authors note that fixed anchors might not match the current scene requirements. Positional extrapolation also fails to stop error propagation once the visible window has already drifted. There is a lack of a mechanism to selectively "look back" at specific, high-quality moments from the distant past to correct the present.

How It Works

LongLive-RAG reframes long video generation as a retrieval-augmented generation (RAG) problem. Instead of relying on a decaying local window, the framework treats the entire history of generated latents as a searchable memory bank.

The architecture operates in three primary stages:

1. Latent Indexing: As the generator completes each block, a lightweight, offline-trained encoder maps the clean latent to a compact 1024-dimensional embedding. These embeddings are stored in a historical search bank alongside their corresponding context entries .
2. Discriminative Retrieval: Before denoising a new block, the model uses the embedding of the most recent completed latent as a query. It performs a cosine similarity search to find the top-$K$ most relevant historical latents. The authors implement a "Window Temporal Delta Loss" to prevent the retriever from being lazy. Without this, the encoder produces nearly identical embeddings for adjacent frames. This causes the $K$ budget to be wasted on redundant, local information .

The loss forces embeddings to capture meaningful temporal changes. This ensures the retrieved context is actually non-local. 3. Context Assembly: The retrieved historical latents are injected directly into the transformer's attention mechanism. They sit alongside the local sliding window and any fixed sinks (special tokens used to stabilize attention). This allows the generator to attend to the original, high-fidelity subject even if the immediate preceding frames have started to degrade .

To ensure the search process is stable, the authors also apply a second-order smoothness penalty during training. This prevents the embedding trajectory from becoming noisy. This noise would otherwise cause the retrieved context to jump erratically between blocks.

Numbers

For an engineer concerned with the "tax" this adds to the production path, the overhead is low. The authors report that for a 120s video rollout, the total retrieval overhead is approximately 490 ms. Most of this cost comes from the latent encoding (480 ms per block). The actual top-$K$ similarity search is negligible at only 10 ms [Table 1]. Since the diffusion denoising process is much more expensive, this is a highly efficient trade-off.

In terms of quality, the results are significant. The paper demonstrates that LongLive-RAG consistently outperforms baselines like $\infty$-RoPE and Deep Forcing across multiple AR backbones. Using the VBench-Long metric, the authors report that LongLive-RAG achieves the best average rank across 30s, 60s, and 120s generations [Table 2]. For example, in 30s generation tests for the Causal-Forcing backbone, the model achieved an average rank of 1.33. This outperformed the base model and other history-handling methods. Qualitative comparisons in confirm these numbers.

LongLive-RAG maintains subject and background consistency where other methods show color shifts or duplicated subjects.

What's Missing

There are a few gaps that a practitioner should consider. First, the framework is strictly additive. It relies on a frozen base generator. This means the final video quality is still bounded by the base model's capabilities. If your underlying diffusion transformer lacks the capacity to render high-quality textures, retrieval will only serve to retrieve those low-quality textures.

Second, the retrieval encoder is trained specifically on the WAN VAE latent space. The authors argue this makes it a "general" framework because many backbones share this latent space. However, it introduces a dependency. If you use a custom VAE, you cannot simply drop this in. You would need to undergo the 7-day data construction and training process described in the paper to build a compatible embedding space.

Finally, the paper focuses heavily on visual consistency. It provides less insight into how retrieval affects complex, multi-object interactions. It is unclear if retrieving a distant "subject" latent might introduce conflicting motion cues. This could happen if the subject's movement in the retrieved frame contradicts the current trajectory.

Should You Prototype This

Yes, if you are building long-form video generation and are currently hitting a wall with identity drift. The implementation is modular. Since it doesn't require retraining the core denoising backbone, it is a low-risk addition to an existing AR pipeline. The overhead is predictable and small enough to fit into most serving budgets.

However, do not attempt to use this as a "fix" for a model that produces poor individual frames. It is a stabilizer, not a creator. If you have a solid base model but struggle with 60s+ consistency, the code is available at https://github.com/qixinhu11/LongLive-RAG and is worth a weekend prototype.

Figures from the paper