LiveBand: Live Accompaniment Generation in the Audio Domain

Abstract: We present LiveBand, a real-time system that generates high-fidelity music accompaniments to live audio input, respecting strict causal constraints. Our method trains a causal transformer generator in the continuous latent space of a pre-trained causal audio autoencoder, using adversarial sequence-level supervision from a discriminator. At each timestep, the generator receives only the causally available mix context and Gaussian noise, and predicts accompaniment latents without access to future mix frames or ground-truth target latents. Training is performed in a single parallel forward pass under causal masking, while streaming inference proceeds autoregressively with a rolling attention state. The model's training and inference computations are matched by design, eliminating teacher forcing and the associated exposure bias. On a multi-instrument music accompaniment benchmark, LiveBand improves over prior work on objective measures of audio quality, beat alignment, and mix adherence, while enabling real-time streaming generation without lookahead into the future on consumer hardware.

• The paper introduces a novel causal audio autoencoder paired with a transformer generator to produce high-fidelity, drift-free accompaniments.
• It leverages adversarial sequence-level supervision and avoids teacher forcing to maintain strict real-time operation and alignment.
• Empirical evaluations on Slakh2100 demonstrate superior beat synchronization, mix adherence, and low latency compared to previous methods.

LiveBand: Sequence-Level Adversarial Supervision for Causal Audio Accompaniment Generation

Problem Setting and Motivation

Live audio accompaniment generation demands models that produce musically coherent, high-fidelity accompaniments for streaming inputs under hard real-time and causality constraints. Previous approaches to accompaniment generation—especially those leveraging causal transformers over discrete neural audio tokens—struggle to avoid error accumulation and alignment drift without lookahead, and suffer from exposure bias due to teacher forcing. LiveBand addresses these issues by reframing both the supervision strategy and sequence modeling paradigm. The method leverages a continuous latent space via a causal audio autoencoder, enabling both high-quality audio synthesis and strict causal operation. Importantly, LiveBand eliminates teacher forcing and exposure bias by training with adversarial sequence-level supervision.

Figure 1: Sequence-level supervision directly rewards globally plausible continuations, while the causal inference setup allows practical, real-time streaming under hard latency constraints.

Architectural Components and Training Methodology

LiveBand decomposes the task into two distinct modules:

1. Causal Audio Autoencoder: A variant of CoDiCodec is retrained for strictly causal operation, producing ∼10 Hz continuous latents ($d_{\text{latent}}=128$) from stereo waveforms at 44.1 kHz. This enables real-time bidirectional compression and expansion of audio without any non-causal context leakage.
2. Causal Transformer Generator (G): At each timestep, the transformer receives the current mix latent, an i.i.d. Gaussian noise vector, and, optionally, instrument-class conditioning. There is no access to ground-truth accompaniments or future mix context. Temporal dependencies are modeled solely via self-attention over causal histories; no teacher-forced targets are injected. Causal masking and scalar attention sinks (FlexAttention) ensure O(1) rollout per step, while query-key normalization and RoPE maintain context stability.

To circumvent exposure bias, training and inference conditions are matched perfectly: the same network receives only causally available mix context and noise, generating the entire sequence via masked parallelism during training, but rolling stepwise at inference.

The adversarial discriminator (D) is a non-causal CNN that sees full mix-accompaniment pairs, scoring real versus generated sequences via R3GAN's relativistic loss and adaptive R1/R2 gradient penalties. Discriminator regularization is adaptively tuned to maintain fixed advantage over the generator, preventing instability across datasets or architectures.

Avoidance of Exposure Bias and Drift

The architecture and learning setup directly tackle error accumulation. Sequence-level adversarial training rewards the overall musical plausibility and adherence of complete generated trajectories, as opposed to per-frame next-step losses which forcibly penalize musically optimal realignments and anticipatory corrections. By operating exclusively with causally available context, and without teacher-forcing ground-truth inputs, the distribution mismatch (exposure bias) between training and streaming inference is nullified. This leads to robust long-horizon streaming without the drift and catastrophic error accumulation observed in classical autoregressive systems.

Empirical Evaluation

LiveBand is evaluated on the Slakh2100 dataset, generating accompaniments conditioned on a real-time mix of stems. Metrics involve both content-oriented and structure-sensitive objectives: Fréchet Audio Distance (VGGish, CLAP), Beat Alignment F1, and COCOLA for mix adherence. Subjective listening tests further assess perceptual quality and fit.

Key experimental findings include:

• Superior Coherence and Alignment: LiveBand achieves higher or comparable scores to ground-truth references in alignment metrics (Beat F1, COCOLA), indicating that learned accompaniments are often more tightly synchronized to the mix than the original human performance.
• No Drift Under Rollout: Contrasting sharply with next-step supervised models, LiveBand exhibits near-zero or negative drift in all benchmarks. This is a bold contradiction of prior claims that strict causality fundamentally limits mix-adherence without lookahead (see e.g. [wu2025streaming]).
• Real-Time Viability: With an end-to-end anticipation (latency) of $\tau=0.093$s per frame, LiveBand completes both generation and autoencoder decoding within the physical streaming budget on consumer hardware, supporting practical interactive deployment.
• Strong Subjective Results: Human evaluators significantly prefer LiveBand over strong SMG baselines across quality, stability, and mix adherence, with $p<0.006$ for all consistency/fit measures.
• Robustness to Architectural Choices: The inclusion of KV attention sinks and adaptive gradient penalty regularization further stabilizes long-form streaming, ensuring that error trajectories and attention patterns remain consistent and drift-free.

Theoretical and Practical Implications

LiveBand establishes that the limitations of strictly causal, real-time music generation are not inherent to the task, but rather reflect an optimization and supervision pathology. Specifically, exposure bias and frame-level supervision penalize precisely the musical corrections and anticipatory behavior necessary for high-quality human-AI interaction. The adversarial sequence-level objective shifts the focus to global musical plausibility, achieving a balance of local error tolerance and long-range structure.

The fully matched training-inference regime, ensured by the generator's input design, provides a strong blueprint for future generative audio modeling under hard streaming constraints. Unlike prior work that relied on expensive self-forcing or post-training iterative rollout, LiveBand's single-pass adversarial training achieves both computational efficiency and streaming robustness.

Speculation and Directions for Future Work

While LiveBand robustly addresses the alignment, coherence, and error accumulation problems, there remains a substantial gap between generated and ground-truth audio quality in terms of perceptual evaluation (FAD). This can be largely attributed to the limitations of current causal autoencoder architectures. Future work should target causal neural codecs with improved rate-distortion properties, potentially via diffusion-based sequence modeling or enhanced discriminative objectives in the latent space.

The framework is also agnostic to the conditioning modality—future extensions could integrate richer context signals (e.g., symbolic scores, text-based description, multimodal video cues) or enable explicit control mechanisms. Furthermore, the sequence-level adversarial training paradigm is immediately applicable to other real-time, causally-constrained generative scenarios, such as interactive speech, video, or multimodal co-generation.

Conclusion

LiveBand demonstrates that strictly causal, real-time music accompaniment generation is feasible without lookahead or teacher forcing, provided the supervision aligns with the structure of the streaming setting. Sequence-level adversarial objectives admit models that are robust to alignment drift, exposure bias, and error accumulation, achieving superior temporal consistency and adherence under practical deployment constraints. This approach provides a new baseline for interactive generative audio systems that require both high fidelity and zero-latency operation, and motivates further research into causal autoencoding and sequence-level adversarial learning for real-time AI generation (2606.03803).