RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling

Can We Stop Treating Every Video Frame Equally?

Video generation models are incredibly slow. They treat every single frame as equally important during the entire denoising process. Even when a video is mostly static, the model spends massive amounts of compute on every frame. It performs heavy 3D spatiotemporal attention (a mechanism linking pixels across space and time) at every single timestep. RhymeFlow attempts to break this redundancy. It picks important "key" frames to process fully. It lets other frames skip steps and uses smart math to fill in the gaps.

The cost of uniform denoising

Current state-of-the-art video models, specifically those built on Diffusion Transformers (DiTs), face a brutal scaling wall. The core issue is the quadratic complexity of 3D spatiotemporal attention. This mechanism allows pixels to attend to neighbors in a frame and counterparts in other frames. As video length or resolution increases, the number of tokens explodes. This makes inference prohibitively expensive.

The industry's current response has been to optimize within the denoising step. Researchers have deployed sparse attention, KV-caching (storing previous attention keys and values to avoid recomputation), and model compression. These make each individual pass cheaper. However, as the authors note, these methods still follow a rigid, synchronous constraint. Every frame in the sequence must undergo the same complete, dense denoising process across all diffusion timesteps. This assumes every frame is equally demanding. That is rarely the case in natural video.

Decoupling the temporal trajectory

The authors ask a specific question: Can we decouple the denoising trajectories of individual frames? They hypothesize that if we anchor the video with pivotal keyframes, the intermediate states of non-keyframes become predictable. These keyframes capture critical semantic shifts (major changes in meaning or objects).

To test this, the researchers developed a framework that moves away from synchronous scheduling [Figure 1a]. Instead of forcing all frames through the same pipeline, RhymeFlow implements an asynchronous schedule [Figure 1b]. The investigation follows a three-part move. First, they use a single-step denoising proxy to estimate "clean" latents. They then identify keyframes via cosine similarity (a measure of how similar two vectors are). Second, they implement a progressive skip strategy. Non-keyframes skip more steps as the process moves from high-noise to low-noise regimes. Finally, they address the problem of broken temporal coherence. They use a latent trajectory projection module to help keyframes attend to "teleported" non-keyframes.

Asynchrony without artifacts

The results suggest the "predictability" hypothesis holds water. Testing on Wan 2.1, the authors report a 1.53× speedup. This uses a default setup of 8 warm-up steps and 4 keyframes. They also improved visual metrics. For example, PSNR (a measure of reconstruction quality) reached 26.291. SSIM (a measure of structural similarity) reached 0.783. These values outperform the original dense model. On CogVideoX-v1.5, they report a 1.78× speedup.

Crucially, this isn't just about skipping work. The latent trajectory projection is indispensable. It uses linear interpolation to estimate missing states. Without it, the model suffers a quality collapse. PSNR drops to 20.630 [Table 5]. The error analysis in confirms the projection works.

It keeps the mean latent error remarkably low, even in high-dynamic videos.

However, there is a limit to the aggression. The authors tested an ultra-aggressive skipping stride ($n_{skip} = 7$). This triggered visible temporal aliasing and "shimmering" artifacts .

This happens because linear projection assumes a smooth ODE (Ordinary Differential Equation) path. When the stride is too large, the model misses non-linear motion.

A new dimension of optimization

If this approach scales, it suggests a new frontier for video acceleration. The goal is making the denoising schedule smarter.

There are three immediate implications for production systems: 1. Orthogonal Gains: RhymeFlow optimizes inter-step efficiency. This means it decides which frames are updated when. This is different from intra-step methods like SAP. SAP optimizes how tokens interact during a single update. Because they are different, they can be combined. The authors show an "Ours + SAP" setup. It reaches a 1.93× speedup on CogVideoX-v1.5. 2. Memory Management: Asynchrony requires a specialized KV-cache. The authors use a per-layer rolling cache. This stores only the two most recent post-attention hidden states. This kept peak VRAM usage at 42.6 GB. This is actually lower than the original dense model's 44.3 GB [Table 6]. 3. Content-Adaptive Inference: This method is content-aware. In a scene with minimal motion, the number of keyframes stays low. In high-action scenes, the system forces more frequent updates.

The obvious follow-up is to test the limits of the "progressive" aspect. The paper uses a piecewise constant stride. A continuous, learned scheduling function might extract more speed. It could potentially avoid the shimmering threshold seen in .

Figures from the paper