TurboServe: Serving Streaming Video Generation Efficiently and Economically

Generating video in real-time is difficult. Users stay connected for long periods. Demand also fluctuates unpredictably. Traditional video generation models typically operate in an "offline" fashion. A user submits a prompt and waits several seconds or minutes. They wait for a complete file to return.

However, a new wave of streaming video generation is shifting this paradigm. It moves toward interactive, progressive synthesis. Video is produced chunk by chunk. This allows users to watch it as it is being created.

This shift introduces a complex infrastructure challenge. Because these sessions are long-lived, the system must preserve the "state." State is the mathematical context and cached data required for continuity. This must persist even when a user becomes momentarily idle. Current serving systems are largely optimized for stateless Large Language Model (LLM) requests. They struggle with this requirement. They often treat each request as an isolated, short-lived task. They fail to account for the persistent nature of a streaming session.

A new study from researchers at Shanghai Jiao Tong University, Tsinghua University, and Shengshu Technology introduces TURBOSERVE. This system is designed to manage the tension between long-lived sessions and volatile demand.

The mismatch of stateless serving

Existing video generation pipelines are mostly optimized for "stateless" jobs. In a stateless model, each request is a fresh start. All resources are released once the job is done. But streaming video generation is fundamentally "stateful." As shown in, a streaming session is a persistent entity. It alternates between active generation and idle periods.

This persistence creates two specific headaches for cluster managers. First, there is "session duration heterogeneity." Some users might generate a five-second clip. Others might engage in a twenty-minute interactive session [, Left].

If a system treats all sessions as interchangeable, it makes poor placement decisions. This leads to "hot" GPUs (overloaded processors) while others sit idle.

Second, "temporal user-demand heterogeneity" occurs when active users spike or drop sharply [, Right]. Static provisioning allocates a fixed number of GPUs regardless of traffic. This forces a brutal choice. You can provision for peak demand and waste money during lulls. Or, you can provision for average demand and suffer high latency during bursts .

A closed-loop approach to scheduling

The authors propose solving this as an online scheduling problem. They jointly manage two variables: session placement and GPU provisioning. TURBOSERVE uses a closed-loop architecture .

Runtime feedback on load and latency continuously tunes the system.

The system relies on two primary controllers working in tandem:

1. The Placement Controller: This component performs "migration-aware min-max rebalancing." It looks for "bottleneck" GPUs (those with the highest per-chunk latency). It then moves active sessions away from them. To ensure this does not cause more harm than good, the authors use a "gain function." This function only permits a migration if the predicted latency reduction outweighs the cost of moving the session's state.

2. The Autoscaling Controller: This component manages the total GPU budget. It uses "load-driven" logic to track target utilization (the percentage of a GPU's capacity being used). If the maximum load across the cluster exceeds a threshold, the system scales out by requesting more GPUs. If the load drops, it scales in.

To make these moves possible without stalling the video, TURBOSERVE implements several optimizations. It uses "coalesced chunk processing" to batch multiple active sessions on a single GPU. This maximizes throughput. It also supports "GPU-CPU offloading." This allows the system to move a session's state to host memory (the computer's main RAM) when a user is idle. This frees up expensive GPU memory for active users. When a user returns, the state moves back to a GPU to resume generation.

Efficiency gains in production

The researchers evaluated TURBOSERVE using real-world production traces from Shengshu Technology. They tested it on clusters with up to 64 NVIDIA B300 GPUs. The results suggest that coordinating these two controllers is highly effective.

The paper reports that TURBOSERVE reduces worst-case per-chunk latency by 37.5% on average compared to baseline configurations. This is vital for the user experience. In a streaming context, a sudden spike in latency causes stuttering or frozen video frames. Simultaneously, the authors find that the system reduces total GPU operating costs by 37.2% on average.

As illustrated in, the benefits of the joint approach are clear.

Simply migrating sessions (Approach 1) helps latency. Simply autoscaling (Approach 2) helps cost. Only the combined strategy (Approach 3) successfully optimizes both. Furthermore, the authors demonstrate that the migration process is lightweight. They report that the overhead for moving a session between GPUs is only 23 to 30 ms [Table 4]. This accounts for only about 2% to 3% of the total per-chunk generation latency.

Limits of the scheduling logic

While the performance metrics are strong, there are practical boundaries. The authors note that the scheduling algorithm could become a bottleneck in extremely large clusters. Currently, scheduling completes in under 15 ms for up to 64 GPUs [, Left]. For much larger environments, they suggest partitioning the cluster into smaller sub-groups.

Additionally, the effectiveness of the autoscaling controller relies on a "volatility-to-parameter mapping." The system uses historical data to decide how to react to changes. The authors show that this method stays within 6.1% of an "offline oracle" (a theoretical ideal with perfect future knowledge). However, it is still a reactive system. There is an inherent delay between observing rising activity and provisioning new GPUs.

Finally, the system's success depends on the "gain function" used during migration. This function balances latency reduction against migration cost ($\eta$). If these parameters are poorly tuned, the system could enter a cycle of "thrashing." This happens when the system spends more time moving sessions than generating video.

Verdict: Ready for the cluster

If you are building a commercial service for interactive, real-time generative video, TURBOSERVE is a relevant blueprint. The research moves beyond asking "how do we make a better model?" It tackles the reality of "how do we run this at scale?" By proving that session migration and autoscaling are necessary, the authors provide a path for reducing infrastructure overhead.

The system is intended for production-grade environments. This is evidenced by its use of real-world traces and its focus on the interplay between latency and cloud rental costs. Code is reportedly available at https://github.com/shengshu-ai/TurboServe. For practitioners, the takeaway is clear. For stateful workloads, the intelligence of your serving layer must be as dynamic as the users it serves.

Figures from the paper