VampNet: Masked Token Music Synthesis

• VampNet is a non-autoregressive masked acoustic token modeling architecture that combines residual vector quantization with bidirectional transformer models for music synthesis.
• It employs a variable masking schedule and parallel iterative decoding, achieving efficient high-quality audio generation in only 12–36 inference passes.
• Its flexible prompting modalities—including inpainting, compression, and looping—enable controlled, deterministic reconstruction along with creative variation.

VampNet is a non-autoregressive, masked acoustic token modeling architecture for high-fidelity, parallel music synthesis, compression, infilling, and guided acoustic variation. It is distinguished by its two-stage design—residual vector quantization tokenization followed by a bidirectional transformer-based masked token generative model—which enables flexible prompting and rapid waveform generation across a broad range of structured musical tasks (Garcia et al., 2023).

1. Architecture: Residual VQ Tokenization and Bidirectional Transformer Modeling

VampNet’s first stage employs the Descript Audio Codec (DAC), a fully convolutional residual vector quantization (RVQ) audio codec. Given input waveform $x$ at $44.1$ kHz, a $D$-dimensional latent $\mathbf{Z}_{t}$ is computed per timestep, which is quantized sequentially by $N$ RVQ codebooks. The quantization procedure is defined by:

$$\mathbf{Z}_{t} \approx \sum_{i=1}^{N} \hat{\mathbf{Z}}_{i,t}$$

where each quantizer $i$ has a codebook of size $C_i$, emitting discrete IDs at a downsampled latent rate ($44.1 \text{kHz}/768 \approx 57 \text{Hz}$). VampNet’s canonical configuration utilizes $N=14$ codebooks—partitioned into $N_c=4$ coarse and $N_f=10$ fine quantizers—yielding an $8\,$kbps compressed representation.

The second stage consists of two bidirectional transformers:

• Coarse model: 20 layers, embedding dimension $E=1280$, 20 attention heads, relative positional encoding, trained to predict subsets of the $N_c$ coarse tokens.
• Coarse-to-fine model: 16 layers, equal embedding dim, tasked with predicting fine tokens conditioned on coarse tokens.

Both are optimized using AdamW ($\beta_1=0.9,\,\beta_2=0.999$), peak learning rate $1\times10^{-3}$ with $10$k warmup steps, $0.1$ dropout, batch size $25$, running within a $72$ GB GPU memory budget. Training runs for $1$M steps (coarse) and $0.5$M steps (coarse-to-fine). The codebook assignment sequence after tokenization enables segment-wise musical representation for all downstream generative tasks.

2. Masked Acoustic Token Modeling: Training and Loss

VampNet employs a variable masking schedule during training, enabling reconstruction of arbitrarily masked subsets of tokens. For a segment token matrix $Y \in \mathbb{Z}^{T \times N}$, with $Y_M$ the masked entries and $Y_U$ the unmasked, the loss is:

$$L(\theta) = -\sum_{y \in Y_M} \log p_\theta(y | Y_U)$$

A cosine-shaped schedule $\gamma(s)$ controls the masking ratio over the course of training; at iteration $t$ out of $T$, retained tokens $k_t = \gamma(t/T)\cdot D$, with $D = T \cdot N$ the total tokens. This scheme conditions the transformer on a continuum of observability, supporting a spectrum of inference-time prompting modalities.

3. Parallel Iterative Decoding and Non-Autoregressive Sampling

At inference, all tokens are initially masked except those explicit in the user-defined prompt. Decoding proceeds over $I \approx 36$ iterations via:

1. Estimate: Forward pass to obtain p.d.f. $p(y|Y_U)$ for every masked token.
2. Sample: Draw candidates $\hat{y}_t$ for each masked token $t$ from $p(\cdot)$.
3. Confidence ranking: Compute $\log p(\hat{y}_t) + \tau_i g_t$, $g_t\sim\mathrm{Gumbel}(0,1)$, $\tau_i$ annealed linearly to $0$.
4. Select: Retain $k_{i+1}$ lowest-confidence tokens for the next round per masking schedule; unmasked tokens pass to $Y_U$.
5. Repeat: Continue until all tokens unmasked or $I$ passes completed.

This parallel, iterative regime achieves near-convergence in Fréchet Audio Distance (FAD) with 36 passes; even 12–24 steps suffice for high quality. Generation is efficient—10 s of audio samples in $\approx 6\,$s (RTX 3090), an order of magnitude faster than comparable autoregressive models.

4. Prompting Modalities and Steering Mechanisms

VampNet’s core innovation is its support for a wide array of prompt types by selective unmasking:

• Prefix (continuation): Unmask initial $L$s, mask remainder—generates forward continuation.
• Suffix (outpainting): Unmask terminal $L$s, mask earlier—generates backward filling.
• Inpainting: Unmask boundary intervals, mask center—model infills mid-sections.
• Periodic: Unmask every $P$th timestep—model upsamples or varies chosen granularity.
• Compression: Unmask only $N_k$ “coarse” codebooks per timestep, mask “fine” tokens—instructs model to decompress from reduced bitrate.
• Beat-driven: Via external beat detection, unmask short windows around each beat—guides stylistic fills preserving rhythmic anchors.
• Looping with variation (“vamping”): Beat or periodic mask within a loop, obtaining non-repeating variations every cycle.

By modulating $P$, $N_k$, and prompt boundaries, the system continuously trades off between deterministic reconstruction and creative, style- and genre-coherent freeform synthesis. This results in granular control over fidelity, detail, and variation for music co-creation or editing.

5. Metrics, Results, and Performance

On a held-out set of 2 k ten-second musical clips, VampNet’s generative and reconstructive performance is assessed by multiscale mel-reconstruction error and Fréchet Audio Distance (FAD). Key findings include:

• Sampling with 36 passes yields minimum FAD, with 12–24 passes approaching similar audio quality.
• Periodic prompting ($P=16$): Reconstructs musical texture at 50 bps preserving style (FAD far below random baseline).
• Beat-driven prompts yield lowest FAD across prompt types, outperforming even prefix-suffix inpainting; effect attributed to “anchors” guiding meter/style.
• Compression plus periodic masks: Faithful reconstructions at $\geq$600 bps; transition to creative generation at $\leq$200 bps (mel error rises but FAD remains close to real audio; noise baselines collapse).
• No formal human listening study is reported, but informal demonstrations indicate preservation of genre, instrumentation, and rhythmic coherence, with natural timbral and rhythmic variation introduced.

6. Implementation Details and Summary

VampNet is trained on a diverse dataset of 797 k music tracks at 32 kHz, resampled to 44.1 kHz. Tokenization uses 14 residual VQ codebooks, with downsampling to 57 Hz (8 kbps bitrate). The transformer models are configured for deep attention (20/16 layers, 1280-dim, 20 heads), with AdamW optimization (1e−3 lr, 10 k warmup, 0.1 dropout, batch size 25). Inference leverages Gumbel-noised confidence ranking, with high initial temperature annealed to zero.

The unified VampNet architecture, combining residual-VQ tokenization, parallel iterative masked decoding, and flexible prompting, delivers rapid, high-quality, genre/stylistic-coherent music generation. Prompt design and mask scheduling furnish precise control over compression, reconstruction, inpainting, looping, and co-creative variation, consolidating these capabilities within a single model framework (Garcia et al., 2023), constituting a significant advance in general-purpose acoustic token modeling for musical synthesis.