LiveBand: Real-Time Audio Accompaniment

• LiveBand is a real-time system that uses strict causality to generate musical accompaniments directly from live audio latent frames.
• It leverages a causal transformer and a pre-trained audio autoencoder to ensure matched training–inference computation and efficient streaming with KV caching.
• Adversarial sequence-level supervision with an adaptive gradient penalty enhances overall musical coherence and audio quality without teacher forcing.

LiveBand is a real-time system for generating high-fidelity music accompaniments to live audio input under strict causal constraints. It operates directly in the audio domain via the continuous latent space of a pre-trained causal audio autoencoder, and trains a causal transformer generator with adversarial sequence-level supervision from a discriminator. At each timestep, the generator receives only the causally available mix context and Gaussian noise, predicts accompaniment latents without access to future mix frames or ground-truth target latents, and supports streaming inference with a rolling attention state. The model’s training and inference computations are matched by design, eliminating teacher forcing and the associated exposure bias (Pasini et al., 2 Jun 2026).

1. Problem formulation and latent representation

LiveBand addresses real-time music accompaniment generation directly in the audio domain under strict causality. The setting assumes an incoming live audio mix, such as a band or backing track including the human soloist, and requires the system to generate one accompaniment stem, such as bass or drums, in real time. The critical constraint is that no future mix frames are available: at model step $t$, only the mix up to time $t$ can be used; there is no lookahead into future audio (Pasini et al., 2 Jun 2026).

Rather than operating on waveforms directly, the system works in the latent space of a pre-trained causal audio autoencoder. Let $\mathbf{m} = (m_1, m_2, \ldots)$ denote the sequence of mix latent frames and $\mathbf{a} = (a_1, a_2, \ldots)$ the accompaniment latent frames to be generated. At streaming step $t$, the model sees the causal mix history $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$ and predicts a future accompaniment frame $a_{t+1+\delta}$, where $\delta \ge 0$ is the anticipation in latent frames. The latent frame rate is approximately $10$ Hz because the autoencoder downsamples by $4096\times$ at 44.1 kHz, and the anticipation in seconds is

$t$0

For the main model, $t$1 s. This anticipation is described as practically required because the model must observe the mix, compute new latents, and decode them to audio before playback; the setup remains strictly causal because it never sees mix frames beyond time $t$2 (Pasini et al., 2 Jun 2026).

The input audio is a stereo waveform at 44.1 kHz corresponding to the mix, encoded into continuous latents $t$3 with $t$4 and frame rate $t$5 Hz. The output is a latent sequence $t$6, decoded back to stereo waveform at 44.1 kHz using the autoencoder decoder. LiveBand also conditions on instrument class, specifically 18 MIDI instrument categories plus a null class, so the target stem can be specified explicitly (Pasini et al., 2 Jun 2026).

This formulation places LiveBand within the broader research program on streaming accompaniment, but with a more stringent insistence on causal operation in the audio domain. Related work on streaming accompaniment formalizes trade-offs in future visibility and output chunk duration, and reports that naive maximum-likelihood streaming training degrades sharply when future context is unavailable (Wu et al., 25 Oct 2025). A plausible implication is that LiveBand’s design choices—continuous latent modeling, adversarial sequence-level supervision, and training–inference equivalence—are intended to address precisely this failure mode.

2. System architecture

The system is composed of three main latent-domain components: a pre-trained causal audio autoencoder $t$7, a causal transformer generator $t$8, and a non-causal convolutional discriminator $t$9 used only during training (Pasini et al., 2 Jun 2026).

The autoencoder is a causal variant of CoDiCodec. Its encoder accepts stereo waveform at 44.1 kHz, has a causal architecture whose receptive field covers only past samples, and downsamples temporally by $\mathbf{m} = (m_1, m_2, \ldots)$0, yielding latent frames at approximately $\mathbf{m} = (m_1, m_2, \ldots)$1 Hz. The latents are continuous, have dimensionality $\mathbf{m} = (m_1, m_2, \ldots)$2, and are normalized to zero mean and unit variance per channel over the training set. The decoder is also strictly causal, reconstructing waveform from past latent frames without seeing future latents. During LiveBand training, the autoencoder is frozen and serves only as the latent space and as the decoder used at inference (Pasini et al., 2 Jun 2026).

The generator $\mathbf{m} = (m_1, m_2, \ldots)$3 is a causal transformer over latent frames conditioned on the mix. At timestep $\mathbf{m} = (m_1, m_2, \ldots)$4, it receives a mix latent frame $\mathbf{m} = (m_1, m_2, \ldots)$5, an independent Gaussian noise vector $\mathbf{m} = (m_1, m_2, \ldots)$6 with $\mathbf{m} = (m_1, m_2, \ldots)$7, and an instrument class embedding. The mix latent and noise vector are concatenated and linearly projected to the transformer hidden dimension, while instrument conditioning is injected via AdaLN-like conditioning. For the main model, the architecture uses hidden dimension $\mathbf{m} = (m_1, m_2, \ldots)$8, $\mathbf{m} = (m_1, m_2, \ldots)$9 transformer blocks, and $\mathbf{a} = (a_1, a_2, \ldots)$0 attention heads with head dimension $\mathbf{a} = (a_1, a_2, \ldots)$1. Each block is pre-norm and includes Adaptive Layer Normalization conditioned on instrument embedding, causal multi-head self-attention with Query–Key normalization and Rotary Position Embeddings, a causal mask plus attention sink, and a SwiGLU feed-forward network. The output is projected back to the latent dimension $\mathbf{a} = (a_1, a_2, \ldots)$2, producing $\mathbf{a} = (a_1, a_2, \ldots)$3 for each position $\mathbf{a} = (a_1, a_2, \ldots)$4 (Pasini et al., 2 Jun 2026).

The discriminator $\mathbf{a} = (a_1, a_2, \ldots)$5 provides sequence-level adversarial supervision in latent space. It takes as input the concatenation along the feature dimension of accompaniment latents and mix latents, yielding a tensor in $\mathbf{a} = (a_1, a_2, \ldots)$6. Its architecture is a non-causal 1D convolutional network with ConvNext-style residual blocks, progressive temporal downsampling across 5 resolution levels, fixed width $\mathbf{a} = (a_1, a_2, \ldots)$7 channels, and no normalization layers, following R3GAN guidelines. Instrument-class conditioning is injected by learned scaling in residual blocks. The output is a single scalar logit $\mathbf{a} = (a_1, a_2, \ldots)$8. Because $\mathbf{a} = (a_1, a_2, \ldots)$9 is used only during training, it is allowed to exploit full-sequence information to assess global realism and mix–accompaniment coherence (Pasini et al., 2 Jun 2026).

A notable architectural property is that the generator never consumes ground-truth target latents. It is always driven by mix context and noise. This distinguishes LiveBand from systems that rely on teacher-forced target histories, and it also distinguishes it from retrieval-based live accompaniment systems such as LyricJam Sonic, which re-contextualizes previously recorded clips rather than generating new accompaniment latents from scratch (Vechtomova et al., 2022).

3. Causality, anticipation, and matched training–inference computation

LiveBand’s causal specification is strict in both the conditioning channel and the output channel. At model step $t$0, the available signals are the mix latents $t$1, past hidden states in the transformer via KV cache at inference, and independent noise $t$2. The unavailable signals are future mix latents $t$3, future accompaniment latents $t$4, and ground-truth accompaniments $t$5, which are never used as input (Pasini et al., 2 Jun 2026).

The generator is trained with parallel causal computation. For a crop of length $t$6 in latent frames, all positions $t$7 are processed in a single forward pass, but a causal attention mask ensures that the hidden state at position $t$8 only attends to positions $t$9. Consequently, the hidden state at $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$0 in the parallel pass is exactly what would be obtained by decoding sequentially with KV cache. At inference, by contrast, the model runs autoregressively: each step ingests one new mix frame $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$1 and one noise vector $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$2, uses cached keys and values for past positions, and outputs $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$3. The KV cache is maintained and potentially truncated, and attention sinks are used to mitigate degradation when cache rolling evicts older entries (Pasini et al., 2 Jun 2026).

The authors emphasize that training and inference computations are “matched by design.” Inputs during training and inference are the same in kind—causal mix context, independent per-step noise, and no target latents—and the same causal mask is used, so the internal computation paths are identical. This removes the discrepancy usually introduced by teacher forcing (Pasini et al., 2 Jun 2026).

The paper states the standard teacher-forcing loss as

$\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$4

with ground-truth history during training, whereas inference samples from

$\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$5

creating exposure bias. LiveBand avoids this mismatch entirely. The generator never consumes its own past outputs nor the ground-truth accompaniments; temporal dependencies are carried internally via attention over the hidden state sequence, while the external per-step inputs remain

$\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$6

plus instrument class, both at training and at inference. The result is that there is no distribution shift between training and inference, and early mistakes do not become conditioning tokens for later predictions (Pasini et al., 2 Jun 2026).

This emphasis on matched computation intersects with broader work on frame-synchronous streaming inference. StreamMUSE, for example, frames live accompaniment as generation synchronized to an external clock in a client–server architecture, and reports a strong coupling between real-time performance and music quality (Zheng et al., 10 Jun 2026). LiveBand differs by centering the causal latent generator itself rather than a networked scheduling protocol, but both systems treat temporal synchronization as a first-class modeling constraint rather than a deployment afterthought.

4. Adversarial latent-space training

Training is purely adversarial in latent space, using the R3GAN framework with an adaptive gradient penalty. Given a real sequence $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$7 and a generated sequence $\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$8, the discriminator and generator objectives are

$\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)$9

$a_{t+1+\delta}$0

with combined gradient penalty

$a_{t+1+\delta}$1

The total discriminator objective is $a_{t+1+\delta}$2, with $a_{t+1+\delta}$3 controlled adaptively (Pasini et al., 2 Jun 2026).

The adaptive gradient penalty is governed by the discriminator advantage

$a_{t+1+\delta}$4

and the update

$a_{t+1+\delta}$5

with step size $a_{t+1+\delta}$6 and target advantage $a_{t+1+\delta}$7. If the discriminator becomes too strong, $a_{t+1+\delta}$8 is increased; if it is too weak, $a_{t+1+\delta}$9 is decreased. The stated purpose is to stabilize GAN training without manual tuning of $\delta \ge 0$0 (Pasini et al., 2 Jun 2026).

A central feature of the training regime is what it omits. There is no reconstruction loss, no feature matching loss, and no explicit beat or rhythm objective. Sequence-level adversarial supervision in latent space is the only learning signal. Because the discriminator sees entire sequences of mix and accompaniment latents, it is expected to reward plausible timbres and audio quality, correct rhythmic alignment with the mix, and harmonic compatibility and global coherence, insofar as the latent space encodes these properties (Pasini et al., 2 Jun 2026).

The paper characterizes this as distribution-level rather than frame-level supervision. This suggests a tolerance for small timing deviations that would be penalized harshly by framewise losses, while still favoring globally coherent accompaniment. A plausible implication is that LiveBand is deliberately optimized for musically plausible continuations rather than for pointwise reconstruction of a single target stem.

This training philosophy diverges from other real-time music systems in the provided literature. ReaLJam uses a Transformer-based agent fine-tuned with reinforcement learning to improve robustness, local coherence, and user-facing jamming behavior in symbolic accompaniment (Scarlatos et al., 28 Feb 2025). LyricJam relies on cross-modal latent alignment and BERT-based ranking to generate mood-congruent lyrics from live instrumental music (Vechtomova et al., 2021). LiveBand instead commits to adversarial sequence-level learning in a continuous audio latent space, with no auxiliary reconstruction objective and no symbolic intermediate representation (Pasini et al., 2 Jun 2026).

5. Streaming inference and real-time operation

At inference, LiveBand runs in streaming mode. Incoming waveform is continuously encoded by the causal autoencoder encoder into mix latents $\delta \ge 0$1 at approximately $\delta \ge 0$2 Hz, one frame every 4096 samples. For each latent-frame step $\delta \ge 0$3, the system draws noise $\delta \ge 0$4, feeds $\delta \ge 0$5 and instrument class to the transformer $\delta \ge 0$6 together with the KV cache containing previous steps, and obtains the predicted accompaniment latent $\delta \ge 0$7. This latent is then passed to the autoencoder decoder to obtain the waveform segment beginning at the corresponding time instant. The paper describes this as one-step generation: each latent is predicted once, with no diffusion sampling loops (Pasini et al., 2 Jun 2026).

The streaming efficiency hinges on KV caching, which avoids recomputing attention over past tokens. Because real systems operate with finite cache, older entries may need to be evicted; the model therefore uses an attention sink, defined as one or more initial positions that remain in the KV cache and act as anchors attracting attention (Pasini et al., 2 Jun 2026). The provided data block truncates the further elaboration of this mechanism, but its stated role is to mitigate performance drop under cache rolling.

The abstract reports that on a multi-instrument music accompaniment benchmark, LiveBand improves over prior work on objective measures of audio quality, beat alignment, and mix adherence, while enabling real-time streaming generation without lookahead into the future on consumer hardware (Pasini et al., 2 Jun 2026). The specific metric values are not given in the provided material, but the qualitative positioning is clear: the system is intended to unify strict causality, real-time deployability, and high-fidelity accompaniment generation in one audio-domain model.

In relation to adjacent research, LiveBand occupies a distinctive point in the design space. “Streaming Generation for Music Accompaniment” formalizes the trade-off between future visibility $\delta \ge 0$8 and output chunk duration $\delta \ge 0$9, and finds that increasing effective $10$0 improves coherence while larger $10$1 improves throughput but degrades accompaniment because of reduced update rate (Wu et al., 25 Oct 2025). StreamMUSE studies frame-synchronous streaming inference in a client–server setting and shows that optimal real-time performance depends on deployment environment and round-trip latency (Zheng et al., 10 Jun 2026). LiveBand’s contribution is different in emphasis: it removes lookahead into future audio entirely, performs generation in a continuous latent audio space, and matches training and inference by construction (Pasini et al., 2 Jun 2026).

6. Position within real-time music generation research

LiveBand belongs to a broader class of systems that treat live musical interaction as an online generation problem, but it differs from several neighboring paradigms in both representation and supervision. LyricJam and LyricJam Sonic are real-time co-creative systems that listen to live music and generate lyric lines, or retrieve and re-contextualize archived audio clips, rather than synthesizing a new accompaniment stem directly from causal audio input (Vechtomova et al., 2021, Vechtomova et al., 2022). ReaLJam addresses low-latency human–AI jamming through a Transformer-based symbolic chord accompanist trained with reinforcement learning and exposed to the user through anticipation and visualization protocols (Scarlatos et al., 28 Feb 2025). AILive Mixer addresses a different live-music problem, namely zero-latency automatic multitrack mixing under stage bleed, using VGGish embeddings, transformers across channels, and one-frame-ahead gain prediction (Zurale et al., 16 Mar 2026). VersBand, despite its “virtual band” framing, focuses on multi-task song generation with prompt-based control and aligned vocals and accompaniment, using flow-based modules rather than strict live causal inference (Zhang et al., 27 Apr 2025).

Against this background, LiveBand’s defining characteristics are its strict causal setup, its audio-domain accompaniment synthesis, and its matched-by-design training and inference computation (Pasini et al., 2 Jun 2026). Unlike retrieval systems, it does not rely on a fixed archive of clips. Unlike symbolic jamming systems, it does not delegate timing and timbre to downstream rendering of symbolic tokens. Unlike offline prompt-driven song generation, it does not assume full access to the input sequence.

The paper’s insistence that the generator never consumes ground-truth target latents and never conditions on future mix frames also addresses a common misconception in streaming generation: that “real time” can be preserved while quietly relying on teacher forcing during training or on short lookahead windows during inference. LiveBand explicitly rejects both. Its anticipation $10$2 is framed not as predictive lookahead into future input, but as a deployment-driven output offset that remains consistent with strict causality because the conditioning stream itself is never allowed to access $10$3 (Pasini et al., 2 Jun 2026).

A plausible implication is that LiveBand functions as a reference design for causal audio-to-audio accompaniment generation under realistic deployment constraints. The provided related literature suggests complementary future directions rather than direct replacements: anticipatory and agentic objectives for negative-visibility streaming regimes (Wu et al., 25 Oct 2025), adaptive synchronization protocols in networked deployments (Zheng et al., 10 Jun 2026), and RL-tuned cooperative behavior for human–AI jamming (Scarlatos et al., 28 Feb 2025). Within that emerging landscape, LiveBand establishes a technically specific answer to the question of how to generate accompaniment in the audio domain, in real time, under strict causality, without teacher forcing and without future context (Pasini et al., 2 Jun 2026).