LVSA: Training-Free Sparse Attention for Long Video Diffusion

Generating long videos with current AI models is a struggle between compute costs and visual quality. As you increase the number of frames, the cost of dense self-attention grows quadratically. This means the computational load increases by the square of the sequence length. Even worse, once you push past the model's original training horizon, the video often "collapses." This results in frozen, repetitive, or looping footage.

Current state-of-the-art approaches attempt to solve this in two ways. They either retrain models for longer contexts or use mathematical hacks that fail to maintain high-quality motion. This paper introduces LVSA (Long Video Sparse Attention). It is a training-free method that allows existing models to generate much longer videos. It does this by only attending to a smart subset of frames. It promises speed and a way to bypass the "frozen video" failure mode.

The Problem

The fundamental issue is the $O(N^2)$ complexity of dense self-attention in Video Diffusion Transformers (DiTs). For a model like HunyuanVideo, generating extra frames pushes the KV (key-value) memory requirements toward the 80 GB limit of a standard GPU.

Even if you have the VRAM, there is a qualitative wall. When a model is trained on 81 frames, attempting to generate 400 frames using dense attention causes the model to lose temporal coherence. As seen in, the video tends to converge to a near-static output.

The subject barely moves across the sequence. Interestingly, the authors note that standard evaluators like VBench-Long actually reward this failure. Because the subject remains consistent, the metric gives it a high score. This "static-rewarding bias" hides the fact that the model has failed to generate meaningful motion.

How It Works

LVSA replaces the dense attention matrix with a sparse pattern. This pattern scales linearly with the number of frames. The architecture relies on three main components:

1. Hybrid Sparsity Pattern: Each query frame attends to two specific sets. It attends to a local temporal window ($W$) of nearby frames to maintain local continuity. It also attends to a set of equidistant global anchor frames ($G$) spread across the entire video. This maintains global structure. This is visualized in .

The "Expanded window" logic ensures the total attention budget per frame remains constant. This happens even when local windows overlap with global anchors. 2. Expanded Window Bounds: A naive window might waste the attention budget when it lands on a global anchor. The authors use an adaptive algorithm (Algorithm 1) to expand the window boundaries. This ensures every frame gets a predictable, uniform amount of computation. 3. Rotating Periodic Global Frames: This is the most critical innovation for quality. If the global anchors were fixed, the model would develop a "fixed-grid bias." In this scenario, frames not in the anchor set become progressively impoverished in representation. To fix this, the authors rotate the global anchor set $G$ at every denoising step (the iterative process of removing noise to create an image). As shown in, every single frame eventually serves as a global anchor over a full cycle.

This prevents the temporal artifacts common in other sparse methods.

Numbers

The performance gains are substantial and scale with video length. The authors report that when using a FlashInfer kernel, LVSA achieves a 3.17× speedup on Wan 2.1 1.3B. It also achieves a 2.98× speedup on Wan 2.1 14B at a 6× horizon compared to dense attention.

Crucially, LVSA solves the Out-of-Memory (OOM) problem for certain architectures. For HunyuanVideo 1.5, dense attention fails at a 2× horizon on a single 80GB GPU. LVSA completes the task while capping peak memory at approximately 60.4 GB [Table 1]. This leaves roughly 19 GB of headroom for the system.

Regarding quality, the authors introduce VQeval to expose "looping" failures. On the Wan 2.1 1.3B model at a 6× horizon, LVSA-FI (using the FlashInfer kernel) achieved a VQeval composite of 60.2. This significantly outperformed the dense attention score of 48.2 [Table 2]. While dense attention's VBench score actually increased as the video became more static, LVSA maintained better imaging quality and genuine motion.

What's Missing

While the results are impressive, there are gaps a practitioner should consider:

• Single-Scene Assumption: The paper explicitly focuses on a single-scene scenario. In production, long videos often involve camera pans or scene transitions. It is unclear how the rotating global anchors handle sudden shifts in visual context.
• Hyperparameter Selection: The effectiveness depends on selecting the right window size ($W$) and anchor period ($T_{per}$). The authors suggest using the model's training budget as a heuristic. However, they do not provide an exhaustive study on how sensitive the results are to these specific values.
• Implementation Requirements: The speedups rely heavily on specialized kernels like FlashInfer. You cannot simply drop this into any standard PyTorch environment and expect these speeds. You need the appropriate backend to realize the reported throughput.

Should You Prototype This

Yes, if you are hitting the VRAM ceiling or seeing "frozen" videos during long-form generation. Unlike many sparse attention papers, LVSA is training-free and model-agnostic. This makes it a low-risk plugin for existing DiT pipelines.

The code is reportedly available at https://github.com/JiusiServe/LongVideoSparseAttention. If you are currently struggling with HunyuanVideo OOMs or Wan model temporal collapse, this is worth a weekend prototype. Just be prepared to tune your window parameters to match your specific model's training characteristics.

Figures from the paper