--- title: 'LiveBand: Real-Time Audio Accompaniment' url: https://www.emergentmind.com/topics/liveband type: topic --- # LiveBand: Real-Time Audio Accompaniment LiveBand is a real-time system for generating high-fidelity music accompaniments to live audio input under strict causal constraints. It operates directly in the audio domain via the continuous latent space of a pre-trained causal audio autoencoder, and trains a causal transformer generator with adversarial sequence-level supervision from a discriminator. At each timestep, the generator receives only the causally available mix context and Gaussian noise, predicts accompaniment latents without access to future mix frames or ground-truth target latents, and supports streaming inference with a rolling attention state. The model’s training and inference computations are matched by design, eliminating teacher forcing and the associated exposure bias [2606.03803]. ## 1. Problem formulation and latent representation LiveBand addresses **real-time music accompaniment generation directly in the audio domain under strict causality**. The setting assumes an incoming live audio mix, such as a band or backing track including the human soloist, and requires the system to generate one accompaniment stem, such as bass or drums, in real time. The critical constraint is that no future mix frames are available: at model step \(t\), only the mix up to time \(t\) can be used; there is no lookahead into future audio [2606.03803]. Rather than operating on waveforms directly, the system works in the latent space of a **pre-trained causal audio autoencoder**. Let \(\mathbf{m} = (m_1, m_2, \ldots)\) denote the sequence of mix latent frames and \(\mathbf{a} = (a_1, a_2, \ldots)\) the accompaniment latent frames to be generated. At streaming step \(t\), the model sees the causal mix history \(\mathbf{m}_{\leq t} = (m_1, \ldots, m_t)\) and predicts a future accompaniment frame \(a_{t+1+\delta}\), where \(\delta \ge 0\) is the anticipation in latent frames. The latent frame rate is approximately \(10\) Hz because the autoencoder downsamples by \(4096\times\) at 44.1 kHz, and the anticipation in seconds is \[ \tau = \frac{\delta}{f_{\text{lat}}}. \] For the main model, \(\tau \approx 0.093\) s. This anticipation is described as practically required because the model must observe the mix, compute new latents, and decode them to audio before playback; the setup remains strictly causal because it never sees mix frames beyond time \(t\) [2606.03803]. The input audio is a stereo waveform at 44.1 kHz corresponding to the mix, encoded into continuous latents \(\mathbf{m} \in \mathbb{R}^{T \times d_{\text{latent}}}\) with \(d_{\text{latent}} = 128\) and frame rate \(\sim 10\) Hz. The output is a latent sequence \(\mathbf{a} \in \mathbb{R}^{T \times d_{\text{latent}}}\), decoded back to stereo waveform at 44.1 kHz using the autoencoder decoder. LiveBand also conditions on instrument class, specifically 18 MIDI instrument categories plus a null class, so the target stem can be specified explicitly [2606.03803]. This formulation places LiveBand within the broader research program on streaming accompaniment, but with a more stringent insistence on causal operation in the audio domain. Related work on streaming accompaniment formalizes trade-offs in future visibility and output chunk duration, and reports that naive maximum-likelihood streaming training degrades sharply when future context is unavailable [2510.22105]. A plausible implication is that LiveBand’s design choices—continuous latent modeling, adversarial sequence-level supervision, and training–inference equivalence—are intended to address precisely this failure mode. ## 2. System architecture The system is composed of three main latent-domain components: a **pre-trained causal audio autoencoder** \(AE\), a **causal transformer generator** \(G\), and a **non-causal convolutional discriminator** \(D\) used only during training [2606.03803]. The autoencoder is a causal variant of CoDiCodec. Its encoder accepts stereo waveform at 44.1 kHz, has a causal architecture whose receptive field covers only past samples, and downsamples temporally by \(4096\times\), yielding latent frames at approximately \(10\) Hz. The latents are continuous, have dimensionality \(d_{\text{latent}} = 128\), and are normalized to zero mean and unit variance per channel over the training set. The decoder is also strictly causal, reconstructing waveform from past latent frames without seeing future latents. During LiveBand training, the autoencoder is frozen and serves only as the latent space and as the decoder used at inference [2606.03803]. The generator \(G\) is a causal transformer over latent frames conditioned on the mix. At timestep \(t\), it receives a mix latent frame \(m_t \in \mathbb{R}^{d_{\text{latent}}}\), an independent Gaussian noise vector \(z_t \in \mathbb{R}^{d_z}\) with \(d_z = 32\), and an instrument class embedding. The mix latent and noise vector are concatenated and linearly projected to the transformer hidden dimension, while instrument conditioning is injected via AdaLN-like conditioning. For the main model, the architecture uses hidden dimension \(d_{\text{model}} = 1024\), \(L = 12\) transformer blocks, and \(H = 8\) attention heads with head dimension \(d_h = 128\). Each block is pre-norm and includes Adaptive Layer Normalization conditioned on instrument embedding, causal multi-head self-attention with Query–Key normalization and Rotary Position Embeddings, a causal mask plus attention sink, and a SwiGLU feed-forward network. The output is projected back to the latent dimension \(d_{\text{latent}}\), producing \(\hat{a}_{t+1+\delta}\) for each position \(t\) [2606.03803]. The discriminator \(D\) provides sequence-level adversarial supervision in latent space. It takes as input the concatenation along the feature dimension of accompaniment latents and mix latents, yielding a tensor in \(\mathbb{R}^{T \times 2d_{\text{latent}}}\). Its architecture is a non-causal 1D convolutional network with ConvNext-style residual blocks, progressive temporal downsampling across 5 resolution levels, fixed width \(C = 1536\) channels, and no normalization layers, following R3GAN guidelines. Instrument-class conditioning is injected by learned scaling in residual blocks. The output is a single scalar logit \(D(\mathbf{a}, \mathbf{m})\). Because \(D\) is used only during training, it is allowed to exploit full-sequence information to assess global realism and mix–accompaniment coherence [2606.03803]. A notable architectural property is that the generator never consumes ground-truth target latents. It is always driven by mix context and noise. This distinguishes LiveBand from systems that rely on teacher-forced target histories, and it also distinguishes it from retrieval-based live accompaniment systems such as LyricJam Sonic, which re-contextualizes previously recorded clips rather than generating new accompaniment latents from scratch [2210.15638]. ## 3. Causality, anticipation, and matched training–inference computation LiveBand’s causal specification is strict in both the conditioning channel and the output channel. At model step \(t\), the available signals are the mix latents \(\mathbf{m}_{\leq t}\), past hidden states in the transformer via KV cache at inference, and independent noise \(z_t\). The unavailable signals are future mix latents \(\mathbf{m}_{>t}\), future accompaniment latents \(a_{>t+1+\delta}\), and ground-truth accompaniments \(a^\star\), which are never used as input [2606.03803]. The generator is trained with **parallel causal computation**. For a crop of length \(T\) in latent frames, all positions \(t=1,\ldots,T\) are processed in a single forward pass, but a causal attention mask ensures that the hidden state at position \(t\) only attends to positions \(\le t\). Consequently, the hidden state at \(t\) in the parallel pass is exactly what would be obtained by decoding sequentially with KV cache. At inference, by contrast, the model runs autoregressively: each step ingests one new mix frame \(m_t\) and one noise vector \(z_t\), uses cached keys and values for past positions, and outputs \(\hat{a}_{t+1+\delta}\). The KV cache is maintained and potentially truncated, and attention sinks are used to mitigate degradation when cache rolling evicts older entries [2606.03803]. The authors emphasize that training and inference computations are “matched by design.” Inputs during training and inference are the same in kind—causal mix context, independent per-step noise, and no target latents—and the same causal mask is used, so the internal computation paths are identical. This removes the discrepancy usually introduced by teacher forcing [2606.03803]. The paper states the standard teacher-forcing loss as \[ \mathcal{L}_{\text{TF}} = -\sum_t \log p_\theta(x_t \mid x_{