DEMON: Diffusion Engine for Musical Orchestrated Noise

Abstract: We present DEMON, a real-time diffusion engine that makes the denoising process playable as a live musical instrument: a control surface both broad (many parameters shaped per-frame across the output) and responsive (each control taking effect as fast as its place in the denoising loop allows). Built on ACE-Step 1.5 and StreamDiffusion's ring-buffer architecture with TensorRT acceleration, it sustains up to 12.3 decoder completions per second for 60-second music on a single consumer GPU (RTX 5090), or 11.3 generations per second at our production ring-depth of 4. At these rates denoising parameters become viable as live performance controls, but the ring buffer propagates per-request changes only at its drain rate, a floor of S denoising steps. We contribute four mechanisms. (1) Per-slot heterogeneous denoise scheduling: each ring-buffer slot owns its timestep schedule, so a moving denoise slider is tracked without wiping the in-flight queue, where the upstream global-schedule design must rebuild and discard it. (2) Shared mutable per-step state, giving any parameter consulted at every solver step next-tick effect, bypassing ring-buffer drain. (3) Per-frame source blending: a sampling-time control on the standard SDE re-noise step, giving a framewise transformation-strength axis that complements scalar denoise scheduling. (4) Windowed VAE decode exploiting receptive-field analysis for an 8.0x decode speedup. Together these separate streaming-diffusion parameters into four propagation classes, by onset and convergence latency.

• The paper presents a streaming diffusion engine, DEMON, that converts denoising parameters into live musical controls for real-time performance.
• It introduces four innovative mechanisms and a windowed VAE decode to achieve low-latency, responsive parameter modulation during music generation.
• Benchmark results show up to 12.3 decoder completions per second, validating DEMON's efficiency and sample-identical quality on consumer GPUs.

DEMON: Diffusion Engine for Musical Orchestrated Noise

Introduction and Motivation

The paper introduces DEMON, a real-time, streaming diffusion engine for musical generation that transforms denoising process parameters into live, expressive controls, effectively creating a playable musical instrument on a consumer GPU. By adapting ACE-Step 1.5—a DiT-based flow-matching music model—to StreamDiffusion's ring-buffer architecture with TensorRT acceleration, DEMON achieves significant throughput (up to 12.3 decoder completions per second for 60-second music) while exposing rich control surfaces for interactive performance. Unlike batch-based or chunked autoregressive systems (e.g., Lyria RealTime), DEMON supports continuous, responsive parameter modulation, with many controls shaped per-frame and propagated efficiently through architectural mechanisms that dissect responsiveness and latency.

Architectural Contributions

Streaming Control Mechanisms

DEMON contributes four core mechanisms that address responsiveness and flexibility within streaming diffusion:

1. Per-slot Heterogeneous Denoise Scheduling enables each ring-buffer slot to maintain its own timestep schedule, preventing queue-wide resets in response to control changes. This allows slider movements to be tracked without discarding in-flight generations.
2. Shared Mutable Per-step State facilitates immediate effect for parameters read at every solver iteration (e.g., per-frame curves), circumventing the ring-buffer drain latency.
3. Per-frame Source Blending introduces a sampling-time control on the SDE re-noise step, permitting framewise transformation strength modulation—orthogonal to global denoise scheduling.
4. Windowed VAE Decode leverages empirical receptive-field analysis of Oobleck VAE to decode only the playback window with overlap margins, achieving an 8.0x decode speedup for sample-identical output.

These mechanisms, together, separate streaming control parameters into four propagation classes—per-request (frozen), migrated schedule (shared-mutable), per-step shared-mutable, and model weights—each with distinct onset and convergence latency characteristics.

System Architecture

DEMON's architecture consists of five main pipeline stages:

• Session API: Handles text encoding, source audio preparation, LoRA management, and model caching.
• StreamPipeline: Implements the ring buffer of staggered denoising stages and concentrates streaming-specific control logic.
• Diffusion Engine: Runs ODE/SDE solvers and per-frame control curves, enabling granular parameter shaping.
• Latent Similarity Filter: Skips costly VAE decode when latents are nearly unchanged, based on deterministic MSE thresholding.
• Windowed VAE Decode: Efficiently decodes only playback windows, eliminating unnecessary full-latent computation.

The use of TensorRT mixed-precision and runtime LoRA refit ensures efficient real-time performance, with strict separation of model components and control mechanisms to maximize composability.

Performance and Latency

DEMON attains robust throughput across multiple operating points on consumer GPUs (e.g., RTX 5090, 4090, 3090), with 12.3 generations/sec at 60s music duration and efficient latency scaling for longer durations (up to 2.6 generations/sec for 240s). The batch scaling is consistently sub-linear, and windowed VAE decode maintains quality while confining latency to the requested playback region.

Figure 1: Decoder TRT engine forward-pass latency at B=1 and B=8 across sequence lengths. The 240s engine (pink) scales linearly up to 240s; 60s engine (blue) plateaus at 1500 frames.

Batch depth enables a throughput-vs-responsiveness tradeoff; higher depth yields better throughput, at the cost of increased per-request control latency. The streaming framework achieves a balance, with depth 4 providing 11.3 generations/sec and ~470 ms per-request control onset latency.

Figure 2: Decoder batch scaling; B=8/B=1 latency ratio is sub-linear at short sequences and approaches linearity at long sequences, framing throughput-responsiveness tradeoff.

Controllability and Control Surface

DEMON's control architecture supports per-frame denoising curves, permitting modulation of multiple parameters—including denoise strength, guidance, velocity scaling, stochastic noise, APG momentum, and target morphing—at 25Hz granularity. This enables expressive real-time manipulation akin to musical performance, surpassing autoregressive and batch-mode paradigms by decoupling control resolution from onset latency.

The per-frame SDE source blending control is empirically demonstrated to:

• Operate independently of schedule truncation.
• Produce consistent per-frame gradients in source preservation across genres and seeds.
• Trade prompt adherence for source fidelity in a controllable, progressive manner.

The x0-target morph further illustrates framewise control toward an independent target latent, with objective measures confirming the modulation's effect across song segments.

Empirical Evaluation

DEMON's streaming pipeline achieves sample-identical output compared to batch-mode generation (16-bit PCM), validating the infrastructural additions as quality-neutral. The SDE control mechanism achieves targeted source preservation with measured trade-off in prompt adherence (CLAP alignment), without degrading distributional quality as measured by FAD against a large music dataset.

The parameter propagation latency is rigorously quantified:

• Per-request changes (e.g., denoise, prompt, source audio) manifest after S ticks (e.g., 649 ms at depth 8, 471 ms at depth 4).
• Per-step shared-mutable parameters (e.g., SDE curve, x0-target morph) take effect on the next tick (e.g., 81 ms), converging progressively.
• Model-weight changes (LoRA refit) are immediate upon runtime refit.
• Heterogeneous scheduling preserves throughput across denoise changes, avoiding dead-air gaps and achieving 100% completion under continuous slider sweep, versus 1.7% for global-reset baselines.

Theoretical and Practical Implications

The delineation of propagation classes maps directly onto practical usability for musicians—responsive controls are confined to parameters read per solver step. Structural parameters bounded at admission to the ring buffer remain tied to the drain floor, defining inherent latency limitations. The architecture's throughput enables a new paradigm for live, expressive music diffusion, but the control is fundamentally over denoising dynamics rather than explicit musical events, highlighting the balance between model distribution adherence and user-driven manipulation.

Evaluative limitations stem from reliance on objective, deterministic metrics (CLAP, FAD, SNR) rather than perceptual studies, particularly regarding the musical compellingness of the instrument; formal listening tests are identified as a critical avenue for future research.

Future Directions

The paper outlines ongoing research paths:

• Distilled VAE Decoder: Lightweight, fast decoder builds to further mitigate decode bottlenecks.
• Latent Channel Semantics: Mapping perceptual effects to VAE channels for targeted manipulation.
• ACE-Step XL Integration: Extending the streaming pipeline to larger, quantized DiT backbones.

Open directions include infinite-length generation, live source encoding, finer beat/melody-level control, real-time perceptual evaluation, and hardware-efficient quantization.

Conclusion

DEMON establishes streaming diffusion models as live, controllable musical instruments with broad and responsive parameter surfaces. The architecture decouples throughput from responsiveness via novel control propagation classes, efficient windowed decode, and composable, low-latency infrastructure. By leveraging ring-buffer pipeline mechanics and per-frame modulation, DEMON offers a framework for expressive real-time music generation and manipulation, with broad applicability and extensibility for future research in neural audio generation and performance tools.