Magenta RealTime: Live Music Generation

• Magenta RealTime is a real-time music generation model that synthesizes high-quality audio using a unified encoder–decoder architecture and chunk-based autoregression.
• It leverages style conditioning through MusicCoCa and SpectroStream tokenization to enable text- and audio-guided control during live performances.
• The system outperforms larger offline models in low-latency scenarios, offering flexible multi-modal control and efficient real-time synthesis.

Magenta RealTime (Magenta RT) is an open-weights, real-time music generation model designed for continuous high-fidelity audio synthesis with explicit human-in-the-loop control. Developed as part of a new class of "live music models," Magenta RealTime enables music creators to steer the model's output in real time via text or audio prompts, implement arbitrary acoustic styles through embedding arithmetic, and flexibly mediate between AI and human audio streams during live performances. This system represents a marked departure from previous generative music approaches, which typically favored offline batch generation or lacked robust mechanisms for continuous user interaction (Team et al., 6 Aug 2025).

1. Modeling Architecture and Tokenization Pipeline

Magenta RealTime employs a pipeline that encodes and generates music at the audio signal level through a sequence of discrete operations:

• Style Conditioning with MusicCoCa: The model incorporates MusicCoCa, a joint embedding model that maps both audio (as log-mel spectrograms processed by a 12-layer ViT) and text (via a 12-layer Transformer) into a shared 768-dimensional latent space. Style conditioning tokens, discretized into e.g., 12 tokens, are derived from this embedding and enable direct manipulation of acoustic style via text, audio, or embedding arithmetic (e.g., interpolating between styles such as "techno" and "flute").
• Audio Discretization with SpectroStream: The raw stereo waveform $a \in \mathbb{R}^{T f_s \times 2}$ is encoded into a stream of discrete tokens using SpectroStream, a codec architecture based on residual vector quantization (RVQ) at a 48 kHz rate. The encoder maps input to a tokenized domain $$\operatorname{Enc}(a) \in \mathbb{V}_c^{T f_k \times d_c}$$, where $f_k$ (e.g., 25 Hz) is the frame rate, $d_c$ the RVQ depth, and $|\mathbb{V}_c|=1024$ is the codebook size. For real-time throughput (targeting ~400 tokens/s and meeting low-latency requirements), only the first 16 RVQ levels are generated during live synthesis.
• Encoder–Decoder Transformer LLM: The core generation model is a single-stage encoder–decoder Transformer trained to predict future audio tokens conditioned on the preceding audio context and the style embedding tokens. Rather than using cascaded or hierarchical models, Magenta RealTime operates on fixed 2-second "chunks" and autoregressively predicts each new chunk from a context of $H=5$ previous coarse chunks (i.e., 10 seconds history). The formal generation objective is:

$$P_\theta(\mathrm{Chunk}_i \mid \mathrm{Coarse}_{i-H:i}, c_i)$$

where $c_i$ is the current style embedding and $\mathrm{Coarse}_{i-H:i}$ are the historical coarse tokens.

2. Architectures for Continuous Real-Time Generation

Magenta RealTime achieves continuous and low-latency synthesis by combining the following mechanisms:

• Chunk-Based Autoregression: Rather than step-wise sequential modeling at the granularity of individual audio frames, the model predicts and streams fixed-length (2-second) discrete audio chunks. Each chunk prediction is based on a sliding window of 10 seconds (i.e., 5 previous chunks), minimizing both computational overhead and error accumulation.
• Coarsened Context for Latency Management: For context windows, only the first 4 RVQ tokens per frame from the acoustic history are retained. This reduces both memory and compute requirements, supporting real time factors (RTF) of 1.8 on an H100 GPU in the T5 Large configuration.
• Strict Causal Generation: All autoregressive generation respects causal constraints, ensuring streaming is strictly forward in time. The chunking and context mechanisms enforce a predictable, bounded delay $D$ on the order of a few seconds, suitable for live music performance.

3. User Control via Text, Audio Prompts, and Audio Injection

Magenta RealTime foregrounds user-guided control at multiple abstraction layers:

• Style Embedding via Prompts: Users can submit control prompts as text, audio, or both. The MusicCoCa module produces the corresponding embeddings for each; these can be combined via a weighted average in embedding space:

$$c = \frac{\sum_{i=1}^{n} w_i \cdot M(c_i)}{\sum_{i=1}^{n} w_i}$$

Here $M(\cdot)$ is the MusicCoCa embedding, $c_i$ the individual prompts, and $w_i$ the prompt weights. This mechanism enables both pure and interpolated stylistic control.

• Audio Injection and Classifier-Free Guidance: Beyond initial conditioning, Magenta RealTime supports a live audio injection loop. Live user audio is periodically mixed with the model's output, forming a combined signal that is re-tokenized and supplied as part of the conditioning context for subsequent chunk generations. To manage the tradeoff between model-driven and user-driven content, classifier-free guidance blending is applied during sampling.
• Synchronous Perception–Action Loop: All control operations occur within a continuous, synchronous perception–action loop between user and model. Strict latency constraints and chunk-based inference ensure that control is both immediate and persistent throughout live sessions.

4. Performance Evaluation and Open-Weights Model Comparisons

Evaluation of Magenta RealTime is carried out against contemporary open-weights music generation models on datasets such as the Song Describer Dataset. The key metrics and results include:

Model	FD_openl3↓	KL_passt↓	CLAP_score↑	#Params	Real-Time Factor
Magenta RealTime	lower	lower	competitive	750M	≥1×
MusicGen Large	higher	higher	similar	3.3B	offline
Stable Audio Open	higher	higher	similar	>1B	offline

• FD_openl3: Fréchet distance between OpenL3 audio embeddings. Lower indicates greater quality and realism.
• KL_passt: Kullback–Leibler divergence of generated vs. reference audio. Lower indicates better match.
• CLAP_score: Audio–text alignment.
• Magenta RealTime outperforms other open-weight systems (lower FD_openl3, lower KL_passt, similar CLAP_score) while using fewer parameters and uniquely supporting sustained real-time operation.

5. System Uniqueness and Technical Distinctions

Several features differentiate Magenta RealTime from prior music generation systems:

• Open-Weights, Strict Causal Streaming: Magenta RealTime is the first model of its kind in the open-weights domain to enable live, infinite music generation with true causal streaming.
• Unified, Efficient LM Design: By condensing both temporal and depth modeling into a single, non-cascaded LLM, the system matches or exceeds the output quality of larger, offline models while maintaining real-time guarantees.
• Flexible Multi-Modal Control: Style embeddings from both text and audio, as well as real-time user audio injection, are manipulated in a unified embedding space for arbitrary prompt mixing and live steering.
• Human-in-the-Loop Facility: The model’s design—latent style mixing, audio injection, and classifier-free guidance—enables responsive, expressive, and interactive control in settings such as live performance, improvisation, or interactive compositional tools.

6. Paradigm Shifts and Comparison to Related Models

Magenta RealTime is part of a broader movement toward real-time, co-creative music AI systems that prioritize continuous user control. In contrast to previous architectures such as ReaLJam—which focuses on melody–chord accompaniment with anticipatory chord planning and reinforcement learning-tuned Transformers (Scarlatos et al., 28 Feb 2025)—Magenta RealTime emphasizes direct control over full audio generation using encoder–decoder LMs with streaming and chunking infrastructure. It thereby avoids the anticipation/commit protocols of symbolic chord prediction in favor of low-latency, continuous audio output guided directly by time-varying user prompts.

Additionally, compared to earlier approaches such as real-time DDSP-based timbre transfer (Ganis et al., 2021) or MusicVAE-driven latent space navigation (Harris et al., 2021), Magenta RealTime generalizes beyond symbolic or timbre-level transformations to encompass arbitrary high-fidelity acoustic generation, controlled at both semantic and signal levels.

7. Technical Resources and Availability

Magenta RealTime is publicly released as an open-weights model, with resources provided to facilitate adoption and further research:

• The full open-source implementation is available for direct integration and experimentation.
• The system forms part of a family of "live music models," including Lyria RealTime, which exposes further control via API access.

This configuration supports a new paradigm for AI-assisted, user-controlled music creation, prioritizing bidirectional interaction, low latency, and high audio realism in live performance environments (Team et al., 6 Aug 2025).