SimVQ Dual Codebooks for Music Tokenization

• SimVQ-based dual codebooks are a discrete representation method that separates vocals and accompaniment via hard routing during quantization.
• The architecture employs a dual codebook design with a shared linear transformation, optimizing high-fidelity audio reconstruction and language model compatibility.
• Empirical results demonstrate improved music tagging, reduced LM perplexity, and superior audio reconstruction at low bitrates.

SimVQ-based dual codebooks are a specialized architectural and training strategy for discrete representation learning in multimodal music tokenization, exemplified by the Duo-Tok system. This method introduces dual, source-aware codebooks quantized via SimVQ (Simple Vector Quantization) and hard routing, explicitly factoring the distinct characteristics of vocals and accompaniment at the tokenization stage. The approach addresses the trade-off between high-fidelity audio reconstruction and LLM (LM) compatibility, advancing state-of-the-art performance in both music representation and generative modeling (Lin et al., 25 Nov 2025).

1. Dual Codebook Architecture and Bottleneck Representation

At the core, the SimVQ-based dual codebook framework extends conventional VQ approaches by maintaining two parallel codebooks: one dedicated to the “vocals” stream and another to the “accompaniment.” After a multi-task SSL encoder is pretrained and fine-tuned, its output for each frame $t$ is a $d$-dimensional vector $h_t \in \mathbb{R}^d$. This vector is then linearly projected into separate embeddings for each branch—vocals and accompaniment—according to pre-computed stem labels (e.g., via a pseudo-Demucs separation). Only the appropriate branch projection processes each frame, resulting in strong source separation at the quantization boundary.

The codebooks are parameterized as matrices $C^h, C^i \in \mathbb{R}^{K \times d}$ (for vocals and instrumental, respectively), where $K$ is the number of entries (in Duo-Tok, $K=32{,}768$). Both codebooks share a single learnable linear transformation $W \in \mathbb{R}^{d\times d}$, yielding the effective codebook $\tilde{C} = C W$ for each branch.

2. SimVQ Quantization: Formulation and Loss Structure

Quantization employs the SimVQ formulation with the shared linear reparameterization:

• Nearest Neighbor Search:

  Each projected bottleneck vector $e_t$ is matched to its closest codebook vector in $\tilde{C}$ by minimizing Euclidean distance:

  $$k^\star(e_t) = \arg\min_{k = 1, \dots,K} \|e_t - \tilde{c}_k\|_2^2;\quad q_t = \tilde{c}_{k^\star(e_t)}$$

• Commitment Loss:

  To ensure codebook stability and smooth encoder–quantizer interaction, a two-term loss is used, leveraging the stop-gradient operator $\text{sg}[\cdot]$: $$L_\text{VQ}(e_t) = \|\text{sg}[e_t] - q_t\|_2^2 + \beta\|e_t - \text{sg}[q_t]\|_2^2$$ The commitment weight $\beta$ is set to 0.25.

• Codebook Update Rule:

  During codebook training, the codebook matrices $C$ are frozen, and only $W$ is optimized. Gradients from $L_\text{VQ}$ and the downstream reconstruction losses flow to $W$, rotating and scaling the codebook space to maximize alignment with the encoder’s output geometry.

3. Training Pipeline and Stagewise Objectives

The Duo-Tok system employing SimVQ-based dual codebooks is trained in a four-stage pipeline, with Stage-3 introducing the dual-codebook quantization. The preceding stages facilitate robust, semantically rich encoder representations, while incorporating musical priors:

Stage	Objective Types	Key Losses	Special Techniques
Stage-1	SSL best-rq-style pretraining	$L_\text{MLM}$ (masked LM)	Masked frame prediction
Stage-2	Multi-task fine-tuning, stabilization	$L_\text{CTC}$, $L_\text{SC}^\text{mel}$, $L_\text{Mag}^\text{mel}$, $L_\text{SC}^\text{chroma}$, $L_\text{Mag}^\text{chroma}$, $L_\text{MSS}$	Gaussian replacement noise ($p=0.2$, $\sigma=1.0$); multi-head supervision
Stage-3	Dual-codebook SimVQ (encoder frozen), source-aware routing	$L_\text{SC}^\text{mel}$, $L_\text{Mag}^\text{mel}$, $L_\text{SC}^\text{chroma}$, $L_\text{Mag}^\text{chroma}$, $L_\text{VQ}$	Hard routing by source label
Stage-4	Latent diffusion decoders for each codebook stream	$L_\epsilon$ (LDM), $L_\text{SI}$ (SI-SNR)	Separate diffusion; SI-SNR boosting

Stage-3 hard-routes each frame to the correct codebook based on vocal/accompaniment labels derived from stem separation. The reconstruction losses focus on Mel spectrogram and chroma fidelity.

4. Hard Routing Mechanism and Codebook Synchronization

Hard routing is performed both during training and inference, using pseudo-Demucs or comparable stem extractors. Each audio frame is labeled as “vocal” or “accompaniment,” with only the relevant branch and codebook involved in quantization and gradient update. The effective algorithmic process is as follows:

for each minibatch of stems do if stem is “vocal” then e = VocalBranchHead(h) use C_voc and W to quantize e → q else e = InstBranchHead(h) use C_inst and W to quantize e → q end compute Mel + Chroma recon. losses from q compute SimVQ loss L_VQ(e, q) update W (and projections, recon. heads) via ∇(L_recon + L_VQ) end

This mechanism effectively ensures the learned codebook representations are specialized and non-interfering across domains, supporting interpretable and source-aligned discrete token streams.

5. Hyperparameters, Optimization, and Scheduling

Training uses the AdamW optimizer with $\beta_1=0.9$, $\beta_2=0.96$, and weight decay of 0.1. Stage-3 adopts a peak learning rate of $1 \times 10^{-4}$, with a 3k step warmup and cosine decay across 30k steps, totaling 100k steps per stage at a batch size of 1,280. SimVQ’s $\beta$ parameter is kept at 0.25 for commitment loss weighting. Gaussian replacement noise in Stage-2 uses $p=0.2$ and $\sigma=1.0$. This configuration empirically supports rapid and stable convergence of both codebooks under hard-routed supervision.

6. Downstream Decoding and Integration with Generative Models

After quantization, frames are replaced by their source-specific discrete embeddings ($q_t$), which are then synchronized into two token streams: $c_\text{voc}$ and $c_\text{inst}$. These sequences condition separate latent diffusion model (LDM) decoders. The system reconstructs audio via embedding and cross-attention of these token sequences into a DiT-style diffusion U-Net, using a composite loss ($L_\epsilon$ for denoising and $L_\text{SI}$ for SI-SNR improvement):

• $z_t = \alpha_t y + \sigma_t \epsilon$
• $$L_\epsilon = \mathbb{E}_{t,\epsilon} \|\epsilon - \epsilon_\theta(z_t, c, t)\|_2^2$$
• $$L_\text{SI} = -(\text{SI-SNR}(\hat{y}_t, y) - \text{SI-SNR}(z_t, y))$$
• $$L_\text{Diff} = L_\epsilon + \lambda_\text{SI} L_\text{SI}$$ where $\lambda_\text{SI}=1.0$

This division ensures that the diffusers remain conditioned on tokens whose semantics reflect source structure, improving both reconstruction quality and LM perplexity in music modeling.

7. Significance, Empirical Performance, and Implications

SimVQ-based dual codebooks instantiated in Duo-Tok demonstrably advance the Pareto frontier of reconstruction fidelity versus LM learnability for music codecs, as evidenced by best-in-class music-tagging average precision, vocabulary-normalized LM perplexity, and state-of-the-art audio reconstruction at 0.75 kbps (Lin et al., 25 Nov 2025). The architecture’s use of bottlenecked dual quantization, robust SSL pretraining, and hard-wired routing promotes highly decomposable and linearly separable token sequences.

A plausible implication is that this framework could generalize to other multimodal or source-separated tokenization domains where source differentiation is semantically meaningful and beneficial for both compression and generation. The explicit design aligns strongly with contemporary trends towards compositional discrete representation in generative modeling, and its demonstrated scalability and tractability with large codebooks suggest future applicability to broader musical and non-musical sequence modeling problems.