DEMON: Diffusion Engine for Musical Orchestrated Noise

Most music generation models are batch-mode tools. You specify a prompt, wait, and listen. While recent autoregressive systems have moved toward streaming, they typically generate audio in fixed chunks. This means control updates—like changing the tempo or brightness—can only happen at chunk boundaries. This creates a massive temporal granularity problem. Musicians expect the sub-millisecond responsiveness of a synthesizer or a DAW.

DEMON aims to turn diffusion models into real-time, playable musical instruments. Instead of waiting for a model to finish a song, DEMON lets musicians change the sound live. It uses a "ring buffer" system—a circular queue of in-flight generations at different denoising stages. This keeps the music flowing smoothly even when you turn knobs or change styles mid-stream.

The Problem

The status quo for streaming diffusion relies on architectures like StreamDiffusion. These use a ring buffer to achieve one output per forward pass. This maximizes throughput, but it introduces a significant responsiveness bottleneck. In these systems, per-request parameters are baked into each slot at submission time. Consequently, a change only propagates once the ring buffer "drains." The system must wait for in-flight generations to complete their full denoising schedules.

If you run a deep buffer to maximize throughput, this "drain floor" can cause latencies of nearly 650ms. Furthermore, many implementations use a single global timestep tensor. If a user moves a slider to change denoising strength, the system must rebuild the global schedule. This wipes the in-buffer and causes audible dead-air gaps. For a live performer, a system that produces audio only 1.7% of the time during a continuous sweep is effectively broken.

How It Works

DEMON solves this by decoupling the ring-buffer depth from the denoising step count. It treats each slot in the buffer as a first-class, state-carrying object. The architecture, illustrated in, breaks parameter propagation into four distinct classes.

1. Per-slot heterogeneous scheduling: Instead of a shared global tensor, each slot owns its own timestep schedule. When a user moves a denoise slider, new slots are admitted with the new strength. In-flight slots continue their original trajectories. This prevents the "queue wipe" seen in prior work. It ensures the output stream remains uninterrupted.
2. Shared mutable per-step state: To bypass the $S$-tick drain floor, certain parameters are held as pipeline-level mutable state. These are consulted at every single denoising step rather than just at submission. This allows for "1-tick onset." A change, like an SDE (Stochastic Differential Equation) source-blending curve, takes effect on the very next forward pass.
3. Per-frame source blending: The engine implements a control on the SDE re-noise step. This allows the user to shape a curve. This curve determines, frame-by-frame, how much the model transforms the audio versus how much it anchors to the original source latents.
4. Windowed VAE decode: The VAE (Variational Autoencoder) decode is often a major bottleneck. The authors exploit the receptive field of the Oobleck VAE. This is the temporal window of context needed to produce a valid sample. They decode only the active playback window with small overlap margins.

Numbers

The authors report impressive throughput on consumer-grade hardware. Using an RTX 5090, DEMON sustains up to 12.3 decoder completions per second for 60-second music at a buffer depth of 8. For their production operating point (depth 4), they report 11.3 generations per second.

The impact of these optimizations is quantifiable. The windowed VAE decode strategy provides an 8.0x speedup compared to a full 60-second decode. This drops latency from roughly 56ms to 7ms. Regarding responsiveness, the authors demonstrate that per-request changes still face the $S$-tick drain floor. However, their heterogeneous scheduling maintains a 100% completion rate during continuous slider sweeps. In contrast, the StreamDiffusion-style global-reset baseline drops to a 1.7% completion rate.

Latency scaling is also documented. As shown in, the decoder TRT (TensorRT) engine latency scales sub-linearly with batch size.

However, it scales linearly with sequence length. At a batch size of 8, the decoder cost for a 60s sequence is approximately 80.6ms. This climbs to 381ms for a 240s sequence. This makes the generation duration a critical choice for managing the "tick" latency of the system.

What's Missing

Despite these performance gains, several engineering and qualitative trade-offs remain.

First, the paper lacks a formal human perceptual evaluation. The authors rely on objective proxies like CLAP (Contrastive Language-Audio Pretraining) alignment and FAD (Frechet Audio Distance). While these are standard, they do not account for the subjective "feel" of the instrument.

Second, the "fast-onset" schedule migration is noted by the authors to produce "transient muffling" in the audio. This mechanism allows for 1-tick onset of denoising changes. The authors admit they lack formal perceptual data to quantify this. They opt instead for the slower, more coherent heterogeneous scheduling for their primary control surface.

Finally, the system is limited by the underlying ACE-Step 1.5 model. It inherits the same genre coverage and musical quality constraints. Furthermore, the DiT (Diffusion Transformer) employs global temporal attention. This means sharp control transitions are naturally smoothed out by the model's internal coupling. You cannot achieve truly independent, frame-perfect control if the model's attention mechanism looks at the whole timeline.

Should You Prototype This

Yes, if you are building interactive audio tools. The core architectural insights are highly actionable. Specifically, decouple your buffer depth from your step count. Use windowed VAE decoding to prevent bottlenecks. The windowed decode alone is a mandatory optimization for any long-form audio streaming system.

If you want the full "playable instrument" experience, focus on the per-step shared mutable state. The ability to bypass the ring-buffer drain for high-frequency controls is vital. This is what separates a batch processor from a real-time controller. Code is available at https://github.com/daydreamlive/DEMON.

Figures from the paper