Multi-Codebook Quantization

• Multi-codebook quantization is a discrete representation method that approximates signals by summing or concatenating selected codewords from multiple specialized codebooks.
• It improves rate-distortion performance and alleviates codebook collapse, making it effective for image compression, neural retrieval, and generative modeling.
• It employs strategies like beam search, residual quantization, and hierarchical assignments to optimize codeword selection for scalable, robust compression in diverse applications.

Multi-codebook quantization is a family of discrete representation learning and compression methods in which a signal is represented by selecting one codeword from each of several codebooks, with the sum (additive models) or concatenation (product/codebook models) of these codewords approximating the original input. This paradigm underpins state-of-the-art approaches in neural data compression, large-scale retrieval, generative modeling, federated learning, post-training quantization, and low-bitrate communication. The multi-codebook approach generalizes classical scalar or single-codebook vector quantization, providing substantially improved rate-distortion efficiency, robustness to codebook collapse, and flexibility in matching task or hardware requirements.

1. Theoretical Foundations and Mathematical Formulation

The canonical form of multi-codebook quantization represents an input vector $x \in \mathbb{R}^d$ as a sum or concatenation of codewords drawn from $M$ distinct codebooks $\{C^{(m)}\}_{m=1}^M$, each with $K$ entries of dimension $d$ or $d/M$ depending on the subspace allocation. For direct-sum models (additive quantization):

$$\hat x = \sum_{m=1}^M e_{c_m}^{(m)}, \quad c_m \in \{1, \ldots, K\}$$

where the integer vector $(c_1, \ldots, c_M)$ serves as a compact code. The encoding problem—finding the code index vector that yields the minimum squared error $\|x - \hat x\|_2^2$—is combinatorial (cost $K^M$ in general), so practical methods employ sequential greedy search, beam search, or differentiable surrogates.

Residual quantization (RQ) and its neural extensions (e.g., QINCo, QINCo2) sequentially quantize the error of previous approximations, using $M$ codebooks each to quantize the current residual, with later codebooks often conditioned on previous reconstructions via a neural network $f_{\theta^{(m)}}$ or context mechanism (Vallaeys et al., 6 Jan 2025, Huijben et al., 2024).

Product quantization (PQ) partitions the input feature into $M$ subspaces, each quantized independently, but does not sum codewords (Yang et al., 2024); hybrid forms (e.g., multi-head codebooks, dual codebook designs) appear in generative models and communication modules (Zhou et al., 2024, Malidarreh et al., 13 Mar 2025).

Mathematically, multi-codebook quantizers minimize expected distortion across a dataset:

$$\mathcal{L}_{\text{quant}} = \mathbb{E}_x\left[ \|x - \hat x(\text{code}(x))\|_2^2 \right]$$

where $\text{code}(x)$ is produced by beam search, classifier-based assignment, or learned assignment networks, depending on statistical and application constraints (Guo et al., 2022, Vallaeys et al., 6 Jan 2025, Wang et al., 27 Oct 2025, Yang et al., 2024, Malidarreh et al., 13 Mar 2025).

2. Methodologies, Architectures, and Optimization

A wide range of architectures instantiate multi-codebook quantization:

• Additive Direct-Sum Quantizers: Each codebook is trained over the full signal space; the final reconstruction is a sum across the selected codewords. No subspace partitioning is used in ASR knowledge distillation frameworks (MVQ-KD), image compression, or neural vector compression (Guo et al., 2022, Vallaeys et al., 6 Jan 2025, Huijben et al., 2024).
• Hierarchical/Residual Quantizers: Sequential quantization, where each codebook approximates the residual error left by prior codebooks—exemplified by residual quantization and recent neural-adaptive extensions (QINCo, QINCo2) (Vallaeys et al., 6 Jan 2025, Huijben et al., 2024).
• Dual/Token-specific Codebook Schemes: Architectures partition features into complementary “global” and “local” halves, each quantized via a separate (often orthogonally updated) codebook; variants include token-specific sub-codebooks or switchable codebook group assignments for structurally diverse data (e.g., faces) (Malidarreh et al., 13 Mar 2025, Wang et al., 27 Oct 2025, Wu et al., 19 Jan 2025).
• Blockwise Quantization: Operand tensors are segmented into blocks, each assigned to one of a small set of codebooks (clustered by block statistics) with blockwise codebook selection and quantization (LO-BCQ) (Elangovan et al., 7 Feb 2025).
• Multi-Codebook Product Quantization for Federated Learning: Each gradient block is quantized with its own codebook, supporting secure aggregation and robust compression under extreme bandwidth constraints (Yang et al., 2024).

Optimization objectives are typically joint over codebooks and, when present, codeword assignment networks or (in neural methods) residual decoders. Training often alternates between assignment and codebook update steps (Lloyd–Max iterations, block clustering), or co-learns all parameters via SGD and straight-through estimators on “hard” codeword selection (Guo et al., 2022, Wang et al., 27 Oct 2025, Vallaeys et al., 6 Jan 2025, Yvinec et al., 2023). Surrogate/proximity gradients mitigate the propensity for assignment tensors to collapse or select extreme values (Yvinec et al., 2023).

3. Advantages Over Single-Codebook and Related Approaches

Multi-codebook quantization offers several core advantages:

• Superior Rate-Distortion Performance: The combination or sum of multiple codewords from different codebooks approximates input vectors with dramatically lower quantization error for a given code length, compared to a single large codebook or scalar quantization. In knowledge distillation for ASR, MVQ-KD achieves performance equivalent to l1/l2 regression losses while reducing storage by $256\times$ (Guo et al., 2022).
• Alleviation of Codebook Collapse: By distributing quantization responsibility, multi-codebook schemes mitigate the under-utilization of codebook entries and collapse modes that arise in single-codebook VQ approaches (Malidarreh et al., 13 Mar 2025, Wu et al., 19 Jan 2025).
• Flexible Compression–Fidelity Trade-offs: The number of codebooks $M$ or codeword size $K$ can be tuned to dynamically trade off between code length, storage, and quantization fidelity (Guo et al., 2022, Zhou et al., 2024).
• Enhanced Robustness and Adaptivity: In federated learning, multi-codebook quantization yields high robustness under non-IID data and maintains accuracy at high compression ratios (Yang et al., 2024). In digital semantic communication, adaptive codebook assignment enables channel-aware operation and up to $3$dB PSNR gain (Shin et al., 16 Apr 2025).
• Efficient and Scalable Training and Inference: Simple nearest-neighbor search, blockwise or sequential Klein search, or classifier-based code assignment enable practical encoding and decoding (Elangovan et al., 7 Feb 2025, Guo et al., 2022, Vallaeys et al., 6 Jan 2025). Variants with lightweight assignment networks or approximators (e.g., pairwise decoders in QINCo2) further accelerate billion-scale retrieval tasks (Vallaeys et al., 6 Jan 2025).

4. Practical Applications Across Domains

Multi-codebook quantization underpins an array of modern applications:

• Knowledge Distillation for ASR: MVQ-KD compresses teacher representations into discrete codes, enabling efficient and accurate student training without massive storage or runtime teacher evaluations (Guo et al., 2022).
• Large-Scale Vector Compression and Retrieval: Residual neural quantization (QINCo/QINCo2) sets state of the art on standard vector search datasets (BigANN, Deep1M), outperforming OPQ/RQ/LSQ in both MSE and recall@1 by wide margins (Vallaeys et al., 6 Jan 2025, Huijben et al., 2024). Pairwise or additive decoders support fast shortlist ranking at billion-scale (Vallaeys et al., 6 Jan 2025).
• Image and Semantic Communication: Multi-head octonary codebooks (MOC-RVQ) and multi-VQ digital communication (ESC-MVQ) match or exceed classical codecs (BPG, JPEG) at a fraction of bandwidth, with architectures mapped efficiently to digital modulation (e.g., 64-QAM), and robust to transmission noise (Zhou et al., 2024, Shin et al., 16 Apr 2025).
• Face/Image Compression: Switchable token-specific codebook quantization (STSCQ) specializes both image-level and token-level codebooks, outperforming global-codebook methods—mean accuracy on facial benchmarks rises by 2-8% at ultra-low bpp (Wang et al., 27 Oct 2025).
• Point Cloud Completion: Dual-codebook guided quantization aligns shallow and deep features, yielding state-of-the-art Chamfer distance and F-score metrics on ShapeNet/PCN (Wu et al., 19 Jan 2025).
• Post-training Quantization for Large Models: Block-clustered quantization iterates between block statistics-based codebook assignment and Lloyd–Max design, achieving sub-1% accuracy loss on W4A4 LLM inference (Elangovan et al., 7 Feb 2025).
• Federated Learning: FedMPQ achieves 90–95% communication reduction compared to uncompressed updates, with $>99\%$ final accuracy retention even under non-IID client data distributions (Yang et al., 2024).
• Compressing Neural Weights and Activations: JLCM (jointly learnable codebooks and mappings) assigns multiple small codebooks via feature clustering and achieves competitive ternary-bit compression on Llama-7B and other large models (Yvinec et al., 2023).

5. Recent Algorithms and Architectures

The contemporary literature demonstrates several algorithmic strategies tailored for high statistical efficiency and hardware practicality:

Approach	Codebook Structure	Assignment/Optimization
MVQ-KD (Guo et al., 2022)	$N$ global codebooks	Classifier initial assignment + local refinement
QINCo/QINCo2 (Vallaeys et al., 6 Jan 2025, Huijben et al., 2024)	Neural, residual	Contextual codeword generation, beam search/MLP
STSCQ (Wang et al., 27 Oct 2025)	Hierarchical token-specific	Routing network + sub-codebooks per group
Dual Codebook VQ (Malidarreh et al., 13 Mar 2025)	Global+Local split	Transformer-updated + deterministic local
LO-BCQ (Elangovan et al., 7 Feb 2025)	Blockwise clusters	Iterative block clustering, Lloyd–Max
FedMPQ (Yang et al., 2024)	Per-feature-block	Federated EMC and periodic k-means
JLCM (Yvinec et al., 2023)	Grouped small codebooks	Joint SGD on codebooks and soft assignments

Crucial implementation tactics include careful codebook initialization (k-means, RQ, cluster-based), dedicated codeword assignment networks, hierarchical or staged training (router pretraining, decoder fine-tuning), and use of straight-through estimators or proximity gradients for stable codeword learning.

6. Empirical Results and Practical Considerations

Multi-codebook quantization methods achieve state-of-the-art performance in both rate-distortion and downstream utility. For instance, QINCo2 reduces BigANN16B MSE by 34% and raises Deep1M8B recall@1 by 24% over prior art (Vallaeys et al., 6 Jan 2025). Dual codebook VQ matches or betters FID scores of much larger codebook VQ-GAN variants on diverse image datasets, despite halved codebook size (Malidarreh et al., 13 Mar 2025). FedMPQ achieves up to 25$\times$ compression in real-world federated setups (Yang et al., 2024). Block clustered quantization yields sub-1% accuracy loss on post-training quantized LLMs at 4.5 bits per scalar overhead (Elangovan et al., 7 Feb 2025).

Best practices involve:

• Matching internal layers by architecture depth (MVQ-KD: teacher at 18th block, student at 9th) (Guo et al., 2022)
• Hyperparameter search on codebook number, size, and fusion weight (e.g., $\lambda$ in KD) (Guo et al., 2022, Malidarreh et al., 13 Mar 2025)
• Beam search width and contextual network capacity tuning for speed–accuracy trade-offs (Vallaeys et al., 6 Jan 2025)
• Robust initialization and re-initialization of rarely selected codewords to ensure dense codebook usage (Yvinec et al., 2023, Vallaeys et al., 6 Jan 2025)
• Exploiting direct codebook-to-constellation mapping in communications to minimize error under digital modulation (Zhou et al., 2024)

7. Extensions, Limitations, and Research Directions

Multiple open challenges persist:

• Data Domain Generalization: Hierarchical and group-based codebook techniques such as STSCQ and MOC-RVQ depend on adequate domain clustering; application to more heterogeneous domains requires either adaptive clustering or soft mixture routing (Wang et al., 27 Oct 2025).
• Efficient Hardware Deployment: LO-BCQ and multi-head schemes are designed for low-overhead decompression and on-chip usage, but large $M$, $K$, or neural codebook approaches stress memory and lookup resources; further hardware-software co-design is critical (Elangovan et al., 7 Feb 2025, Zhou et al., 2024).
• Dynamic Rate Adaptation: Channel-/context-adaptive codebook selection (e.g., ESC-MVQ) and dynamic residual quantization afford bandwidth/accuracy trade-offs; deeper integration with modulation/power allocation is likely to continue (Shin et al., 16 Apr 2025, Zhou et al., 2024).
• Codebook Collapse and Utilization: Even with distributed quantization, insufficient entropy or improper initialization can lead to partial utilization. Transformer-based and residual neural updates mitigate but do not fully solve this for arbitrary data distributions (Malidarreh et al., 13 Mar 2025, Vallaeys et al., 6 Jan 2025).

In summary, multi-codebook quantization constitutes a foundational and increasingly versatile class of quantization schemes that combine high rate-distortion efficiency, structural flexibility, and strong empirical performance, driving progress in compression, search, generative modeling, model deployment, and communication under real-world constraints (Guo et al., 2022, Wang et al., 27 Oct 2025, Vallaeys et al., 6 Jan 2025, Elangovan et al., 7 Feb 2025, Yang et al., 2024, Yvinec et al., 2023, Shin et al., 16 Apr 2025, Zhou et al., 2024, Malidarreh et al., 13 Mar 2025, Wu et al., 19 Jan 2025, Huijben et al., 2024).