Multi-Codebook Fusion Strategies

• Multi-Codebook Fusion is an architectural strategy that employs multiple discrete codebooks to capture diverse representations and enhance generative model fidelity.
• It fuses outputs from independent quantizers via concatenation or summation, enabling improved performance in low-latency audio and high-fidelity image tasks.
• MCF techniques utilize both pre-trained and joint training approaches to optimize code utilization and prevent codebook collapse in complex models.

Multi-Codebook Fusion (MCF) refers to architectural strategies in representation learning and generative modeling that employ multiple discrete codebooks in parallel, with their outputs fused—by concatenation, summation, or other operations—at critical points within the model architecture. This approach aims to improve representational richness, codebook utilization, and downstream fidelity while enabling lower latency for autoregressive or streaming generation tasks. MCF has been adopted in both audio and image domains, including low-latency spoken language modeling and high-fidelity image reconstruction (Wang et al., 13 Nov 2025, Malidarreh et al., 13 Mar 2025).

1. Architectural Fundamentals of Multi-Codebook Fusion

In MCF, the input (such as a waveform or image) is encoded into multiple streams of discrete codes via separate quantizers ("codebooks"), each handling a subset of the latent representation. For instance, in the VocalNet-M2 model, the XY-Tokenizer produces eight streams of quantized speech tokens, each derived from an independently pre-trained codebook, where each codeword is of embedding dimension $D$ (e.g., $D=1024$).

In the domain of image modeling, the Dual Codebook VQ model splits the latent feature map produced by the encoder into "global" and "local" halves along the channel dimension. Each half is quantized with an independent codebook: the global branch integrates information via a 6-layer Transformer encoder before vector quantization, while the local branch uses a deterministic nearest-neighbor assignment to codebook vectors (Malidarreh et al., 13 Mar 2025).

2. Fusion Mechanisms and Mathematical Formalism

MCF architectures use specific fusion mechanisms to integrate multi-codebook representations:

• Audio (VocalNet-M2): Fusion is staged. Semantic (text) context is fused with LLM hidden states by channel-wise concatenation and linear projection, yielding $h^{fused}$. At each audio decoding step $t$, the fused semantic context, upsampled and zero-padded to match audio token rate, is summed with the embeddings from all eight audio codebooks to form $X_t$:

$$S_t = \sum_{j=1}^8 \text{Emb}(a^{cbj}_{1:t}), \qquad X_t = h^{up}_{1:t} + S_t$$

The decoder processes $X_t$ and emits logits through parallel output heads.

• Image (Dual Codebook VQ): The fusion is a strict channel-wise concatenation:

$$z_q = [z_q^g; z_q^\ell], \qquad z_q \in \mathbb{R}^{C \times h \times w}$$

where $z_q^g$ and $z_q^\ell$ are the quantized outputs of the global and local codebooks, respectively.

These operations combine complementary information while maintaining codebook independence during quantization.

3. Codebook Learning and Update Strategies

Two primary paradigms for codebook learning are implemented in MCF systems:

• Pre-trained and Frozen (VocalNet-M2): Codebooks are learned via minimizing reconstruction loss (e.g., on mel-spectrograms) during a pre-training stage. After convergence, the codebooks are frozen, and the downstream model (e.g., the Talker in VocalNet-M2) operates exclusively on the discrete codes.
• Joint Training (Dual Codebook VQ): Both codebooks are trained from scratch with no pre-training. The global codebook updates its entries by backpropagation through the Transformer encoder, ensuring code vectors capture contextual (long-range) dependencies, while the local codebook entries are pushed closer to their assigned local features via the commitment loss. This dual update regime promotes a near-uniform utilization of all code vectors, avoiding the "codebook collapse" observed in single-codebook scenarios (Malidarreh et al., 13 Mar 2025).

4. Training Objectives and Optimization

• Audio Modeling (VocalNet-M2): The Talker model is trained via standard cross-entropy losses applied to all eight parallel codebook streams:

$$L_{talker} = - \sum_{t=0}^{M-1} \sum_{j=1}^8 \log P(a^{cbj}_{t+1} | X_t)$$

With multi-token prediction (MTP), additional supervisory signals for future tokens up to horizon $N_{mtp}$ are incorporated:

$$L_{mtp} = - \sum_{n=0}^{N_{mtp}} \sum_{t=0}^{M-1} \sum_{j=1}^8 \log P(a^{cbj}_{t+n+1} | X_t)$$

The total loss is $L = L_{talker} + L_{mtp}$.

• Image Modeling (Dual Codebook VQ): The loss is the sum of pixel-level reconstruction, parallel vector quantization losses for global and local branches, and an adversarial GAN component:

$$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{rec}} + \sum_{b\in\{g,\ell\}} \left( \|\mathrm{sg}[z_e^b] - z_q^b\|_2^2 + \|\mathrm{sg}[z_q^b] - z_e^b\|_2^2 \right) + \lambda \mathcal{L}_{\mathrm{GAN}}$$

where $\lambda$ is an adaptive weight balancing reconstruction and adversarial gradients (Malidarreh et al., 13 Mar 2025).

5. Architectural Modifications for Multi-Codebook Inputs

To support MCF, decoder architectures are adapted:

• In VocalNet-M2, the Transformer decoder is implemented with "multi-track" output heads, each responsible for predicting the next token of one codebook. Inputs at each timestep are formed by summing all codebook embeddings and upsampled semantic context.
• Input fusion is implemented via linear layers for merging embeddings, and upsampling adaptors are used (e.g., repeat-and-pad ×3) to align semantic and audio token rates.
• In image models, fusion is realized via channel-wise concatenation, enabling the decoder to utilize both global and local quantized representations, with no increase in compute versus a single-codebook design if channel width is held constant (Malidarreh et al., 13 Mar 2025).

6. Empirical Outcomes and Quantitative Analysis

MCF architectures show empirical improvements in both speech and image domains.

Model Variant	First-chunk Latency (ms)	WER (%)	UTMOS	ADE20K FID	CelebA FID	COCO FID
Single-codebook S³	725 ± 9	3.73	4.35	–	–	–
Multi-codebook (8-cb)	405 ± 6	10.43	4.08	–	–	–
Multi-codebook + MTP	349 ± 3	6.07	4.31	–	–	–
Dual Codebook (image)	–	–	–	17.03	4.19	4.19
VQ-GAN (single, image)	–	–	–	22.04	5.66	14.45
VQ-VAE (single, image)	–	–	–	116.85	36.08	86.21

• In spoken language modeling, MCF with MTP (VocalNet-M2) halves first-chunk latency from 725ms (single-codebook) to 349ms while achieving UTMOS ≈ 4.3 (Wang et al., 13 Nov 2025).
• In image reconstruction, Dual Codebook VQ achieves FID improvements of 3+ points on ADE20K (17.03 vs. 20.25 or 22.04) and over 5 points on COCO, despite using half the codebook size (512 vs. 1024 or 6207) (Malidarreh et al., 13 Mar 2025).
• MCF avoids codebook underutilization (collapse) by specialized update mechanisms, resulting in nearly 100% utilization in both global and local codebooks.
• Increasing the depth of MTP layers in VocalNet-M2 reduces WER from 8.56% (no MTP) to an optimal 6.07% (four layers), with only minor UTMOS variation, indicating improved local dependency modeling across codebooks.

7. Comparative Analysis and Broader Significance

Compared to single-codebook approaches, MCF frameworks address several recurrent problems:

• Under-utilization (collapse): Single-codebook VQ models such as VQ-GAN often employ only a small subset of available code vectors, reducing effective representation capacity. MCF (with global/local) maintains high utilization rates in all sub-codebooks by distinct code assignment and update protocols (Malidarreh et al., 13 Mar 2025).
• Latency and Efficiency: Parallel codebook decoding, especially when combined with strategies such as MTP, significantly reduces latency in streaming applications without flow-matching bottlenecks, as demonstrated in VocalNet-M2.
• Representational Richness: By separating long-range context modeling (global) from fine-grained fidelity (local), MCF architectures achieve better tradeoffs between global structure and local detail, as evidenced by improvements in FID, PSNR, and qualitative reconstructions.

A plausible implication is that multi-codebook fusion generalizes as a blueprint for scalable, loss-robust, and computation-efficient representational bottlenecks for both sequential and spatial input modalities. Continued evolution of MCF may yield further advances in codebook expressiveness and modularity across generative modeling applications.