SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer

Achieving real-time, high-resolution video editing—where you can swap a background or change an outfit in a live broadcast—requires balancing massive throughput with strict temporal consistency. Currently, most video-to-video (V2V) models are either too slow for interactive use or suffer from "flicker." This happens when the edit drifts or jumps unnaturally between frames. Researchers have developed SANA-Streaming to solve this. They combine specialized neural architectures with hardware-aware optimization.

The Problem

The fundamental tension in streaming video editing is between memory efficiency and local precision. Standard softmax attention allows every token to attend to every other token. This provides excellent local detail and precise correspondence to the source video. However, as video length increases, the KV cache (the memory used to store keys and values for attention) grows quadratically. This quickly exhausts the VRAM (video random-access memory) of even high-end consumer GPUs.

Conversely, linear attention models are highly efficient for streaming. They compress history into a fixed-size recurrent state (a compact memory that summarizes past frames). The downside is a loss of granularity. As shown in, linear attention maps tend to be dense and diffuse.

They lack the sharp focus on local neighborhoods that softmax attention provides. In practice, this leads to "chunk-to-chunk" appearance jumps. This causes temporal flickering, where the subject's identity or the environment's texture shifts abruptly at the boundary of every processed segment .

How It Works

SANA-Streaming addresses this through a system-algorithm co-design centered on three pillars:

1. Hybrid Diffusion Transformer: Instead of choosing between linear and softmax attention, the authors interleave them. Most blocks use Gated DeltaNet (GDN)—a type of linear attention that uses a delta-rule update to act as a correction-based memory. This maintains a compact global state that does not grow with video length. Periodically, the model inserts softmax attention blocks. These use a sliding window and a "sink" token (a persistent visual anchor) to refine local details .

2. Cycle-Reverse Regularization: To solve the lack of paired long-form training data, the authors implement a unique training loop. The model performs a forward edit. Then, it is tasked with a "reverse edit." It must reconstruct the original source frames from the edited ones using a reverse prompt .

This forces the model to learn a mapping that preserves the underlying structure and motion. This teaches temporal consistency without needing ground-truth edited long videos.

1. Hardware-Aware Optimization: The system is optimized specifically for the NVIDIA Blackwell architecture. This includes custom Triton kernels (software programs that run on GPUs) for GDN. These kernels separate spatial reduction from temporal recurrence. This allows the recurrent state to stay in on-chip SRAM (fast, local memory). Additionally, they use an AutoML-inspired Mixed-Precision Quantization (MPQ) search. This assigns different precisions (FP4, FP8, or BF16) to different layers based on their sensitivity to noise .

Numbers

The headline result is that SANA-Streaming achieves 24 end-to-end FPS (frames per second) at 1280×704 resolution. This runs on a single consumer-grade RTX 5090 GPU. This speed is sufficient for real-time interactive video. Within the model itself, the DiT (Diffusion Transformer) core runs at 58 FPS.

The authors attribute these gains to their system co-design. Compared to a standard BF16 baseline, their implementation of fused GDN kernels and MPQ delivers a 1.59× speedup in DiT latency [Table 2]. This means the core model processes frames significantly faster than the standard precision setting. On the quality front, the paper reports that the model outperforms existing SOTA (state-of-the-art) methods on the OpenVE-Bench. It achieves an average score of 2.27 across various editing categories [Table 1]. Even after distilling the model to just 4 steps for speed, it maintains competitive performance.

What's Missing

While the throughput numbers are impressive, there are gaps in the evaluation. First, the paper acknowledges that the scarcity of high-quality long-video editing samples remains a bottleneck. While Cycle-Reverse Regularization helps, the model may still struggle with complex semantic shifts. These are areas where the "reverse" logic might not generalize perfectly.

Second, the "real-time" claim is heavily dependent on the Blackwell architecture. While they provide runtime breakdowns for older architectures like the A100 and H100 [Table 6], the peak performance is optimized for the RTX 5090. If your production stack uses older chips, you won't see the same advantages from their NVFP4 quantization strategy. Finally, the paper does not deeply investigate how the system handles extremely high-motion scenes. Rapid camera pans often trigger the worst-case failures in causal, recurrent models.

Should You Prototype This

Yes, if you are building interactive video tools for live streaming or gaming. If you have access to modern NVIDIA hardware, this is a major milestone. Moving V2V editing from offline rendering to live interaction is now possible on a single consumer GPU. The hybrid architecture approach is very pragmatic. Using linear attention for the heavy lifting and softmax for the "polish" bypasses the KV cache bottleneck.

Code is reportedly available; see the project page at https://nvlabs.github.io/Sana/Streaming for the canonical link. If you are struggling with temporal drift in your own streaming implementations, the Cycle-Reverse Regularization strategy is worth a prototype.

Figures from the paper