StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration

StreamChar: Real-Time Long-Horizon Streaming Audio-Video Generation

StreamChar is a new system that creates long videos of characters talking in real-time. It uses a smart "orchestrator" to plan speech. A specialized generator ensures the video does not drift or look strange over many minutes of talking.

Generating believable digital humans in real-time is a difficult balancing act. The model must follow a long script accurately. It must also maintain a consistent visual identity. Finally, it must finish its math fast enough to stay ahead of the playback clock. Current models usually fail at one of these. You might get high-quality clips that take minutes to render. Alternatively, you might get fast streaming videos that lose connection to the script. These videos often suffer from "visual drift," where the character's face or clothes morph unexpectedly.

The Question

Researchers at Tongyi Lab addressed a tension in multimodal generative modeling. How do you manage long-horizon coherence in an autoregressive (generating piece by piece) streaming setting? They also needed to maintain the low latency required for interactivity. The core problem is that standard pipelines use a single backbone for everything. This backbone must handle high-level semantic reasoning and low-level spatiotemporal denoising (removing noise from video and audio signals). When these tasks compete for model capacity, errors cascade. The audio drifts from the text. The video drifts from the speaker's identity.

Why The Old Answer Was Incomplete

Previous approaches used monolithic Diffusion Transformers (DiT). These are large backbones that handle everything from text understanding to pixel synthesis. While effective for short clips, they fall apart during long-form streaming. In a chunk-wise setting, local errors in one chunk become the context for the next. This creates a feedback loop. Minor misalignments in lip-sync or facial geometry compound exponentially over time.

Engineers also use aggressive distillation to achieve real-time speeds. Distillation compresses a slow, high-step diffusion model into a fast, few-step student model. However, the authors note this often triggers "mode collapse." This is when the student model loses spatial diversity. The character might talk fast but look like a repetitive, uncanny mannequin.

What They Did

The authors proposed a decoupled architecture that separates "thinking" from "drawing." As shown in, the system uses an LLM-based Orchestrator for long-range planning.

This Orchestrator reads the transcript and historical context. It produces frame-aligned audio conditions ($c_a$). These tell the generator exactly what the acoustic intent should be for the upcoming chunk. This offloads script-tracking from the heavy-duty DiT.

The DiT focuses on short-window, bidirectional denoising. To prevent "drifting" during long rollouts, the team used two mechanisms. First, they implemented a "sink-frame" mechanism. The very first chunk generated acts as a persistent visual anchor .

All subsequent chunks attend to this anchor to maintain identity. Second, they developed a "Progress-Aware Pointer" (PAP). This module predicts exactly where in the transcript the current audio chunk ends .

This allows the system to precisely truncate the transcript during training. This ensures the model stays aligned with the spoken words.

To bridge the gap between teachers and students, they used a two-stage distillation pipeline. Stage I uses Distribution Matching Distillation (DMD) to compress the sampler into a 4-step generator. Stage II performs "online rollout" fine-tuning. In this stage, the student is trained on its own consecutive outputs. This simulates the actual streaming experience.

What They Found

The results suggest that decoupling the architecture is highly effective. The authors report that StreamChar runs in real time on a single H100 GPU. The total per-chunk latency is approximately 1.34 seconds. This fits within the 1.38-second playback budget for 33 frames at 24 fps.

The impact of the sink-frame mechanism is significant. Without the sink chunk, "Quality Drift" increases from 0.0067 to 0.0304 [Table 2]. This metric measures how much the video degrades over time. Qualitatively, as seen in, removing this anchor causes color shifts.

It also causes repetitive, low-diversity spatial behaviors.

Regarding speech accuracy, the distilled student maintains a Word Error Rate (WER) of 3.65% [Table 1]. This represents the percentage of words incorrectly transcribed. This low error rate shows the LLM Orchestrator preserves phonetic alignment. Even after aggressive step reduction, the model stays accurate. Unlike some streaming baselines, StreamChar maintains high motion diversity. It achieves a VBench Dynamic score of 1.0 [Table 2]. This means the character's movements remain varied and natural rather than becoming stiff.

What This Changes

This decoupled orchestration pattern sets a new blueprint for long-form multimodal generation. The primary takeaway is that "intelligence" and "fidelity" should not share the same weight matrix in an autoregressive loop. By separating them, you can optimize the planner for context. You can also optimize the denoiser for speed. Neither task will sabotage the other.

There are two implications for production systems. First, the "sink-frame" suggests a cheap way to fight identity drift. Maintaining a high-fidelity anchor from $t=0$ is more effective than building infinite memory into a transformer's KV cache (a memory storage mechanism for attention). Second, the two-stage distillation recipe provides a clear path to deployment. It moves models from high-quality research versions to low-latency services.

The next step for this architecture is testing scalability. Researchers could use much larger LLM orchestrators. They could also test how it handles non-linear scripts, such as interactive dialogue.

Figures from the paper