Grouped Codebooks: Adaptive & Modular Quantization

• Grouped codebooks are structured partitions of codewords that enable improved vector quantization by separating codewords into distinct, adaptive groups.
• They leverage group-wise optimization, product structures, and input-adaptive methods to mitigate codeword collapse and ensure efficient representation learning.
• Applications span deep generative models, image restoration, communications, and model compression, offering flexible, interpretable, and high-fidelity performance.

Grouped codebooks are structured collections of code vectors or codewords partitioned into distinct subsets, each serving specialized roles within vector quantization, deep generative modeling, communications, or other signal processing tasks. This architecture enables flexibility, improved codebook utilization, modularity, adaptive encoding, and interpretable representations across domains including neural network quantization, CAD sequence generation, spatial genomics group testing, and product quantization for MIMO systems. Grouped codebook designs share the underlying principle that the global set of possible codewords is split into multiple “groups,” with the grouping designed to reflect statistical or semantic structure in the data or to facilitate efficient search, adaptation, or quantization.

1. Formal Definitions and Core Principles

A grouped codebook is defined by partitioning a global set of codewords into $G$ disjoint groups or sub-codebooks: $$C = \bigcup_{g=1}^G G_g, \quad G_g \cap G_{g'} = \emptyset \text{ for } g \neq g'$$ Each group $G_g$ contains $K_g$ codewords. Assignments or quantization within vector quantized models, compression systems, or signal processing pipelines are then based on explicit or implicit selection among these groups.

This structure generalizes classical single-codebook vector quantization, supporting product quantization (Cartesian products of lower-dimensional codebooks), input- or context-adaptive selection, or disentanglement of distinct semantic or statistical sources in the data. In grouped codebook systems, the encoding typically selects both a group index and (often jointly) an intra-group codeword, either deterministically or adaptively, via auxiliary modules or attention mechanisms.

Grouped codebooks can be static (fixed partitioning), dynamically controlled (input-adaptive), or even resampled after training, depending on the application and underlying optimization algorithms (Zheng et al., 15 Oct 2025, Liu et al., 2023).

2. Construction and Optimization Algorithms

The construction and optimization of grouped codebooks are domain-dependent and can be exemplified by several canonical methodologies:

a. Group-Wise Vector Quantization (VQ):

Grouped VQ partitions the codebook and independently optimizes each group, with the grouping parameter $G$ controlling the tradeoff between codebook utilization and reconstruction quality. Updates to each code vector are restricted to samples assigned to its group, and group-wise optimization mitigates codebook collapse (i.e., the underutilization of codewords) ubiquitous in joint or vanilla VQ. The loss function is additive over groups, leading to group-specific gradient flows (Zheng et al., 15 Oct 2025).

b. Product Codebooks via Cartesian Product Structure:

In full-dimension MIMO or tensor decompositions, codebooks are constructed as a product of smaller factor codebooks, e.g., along horizontal and vertical dimensions: $F_{i,j}=(C_{v,i}\otimes C_{h,j})^*$ where $C_{h,i}$ and $C_{v,j}$ are centroids in their respective sub-manifolds, determined via Grassmannian $K$-means clustering on the factor spaces (Bhogi et al., 2021). This factorizes the search over the codebook, reduces complexity, and enables adaptive refinement along each physical axis or functional modality.

c. Input-Adaptive and Disentangled Codebooks:

Frameworks such as AdaCode (Liu et al., 2023) and SkexGen (Xu et al., 2022) learn several basis or semantic codebooks (e.g., for different image components or CAD factors), and combine them via input-dependent mixing weights ($W_{p,k}$) or by fixing and sampling among groups (for controllable generative processes). In these schemes, codebook construction proceeds by jointly learning the discrete embeddings and the mechanisms for adaptive group selection or code-mixing.

d. Grouped Codebooks for Model Compression:

In jointly learnable codebooks and mappings (JLCM), neural network weights are grouped (often by clustering neurons with similar statistics), with each group associated with its own codebook or shared scale factors. Both codewords and assignments are updated jointly, subject to a composite loss that aligns quantized and floating-point networks both in weight and activation space. Proximal search techniques are deployed to stabilize codeword assignment optimization and achieve finer-grained quantization (Yvinec et al., 2023).

3. Applications in Deep Models, Communications, and Structured Data

Grouped codebooks are deployed in a variety of modern computational and signal processing contexts:

Application	Grouping Principle	Representative Paper
Deep generative modeling (VQ-VAE, GANs)	Grouped VQ for codebook utilization and collapse mitigation	(Zheng et al., 15 Oct 2025)
Image restoration and enhancement	Basis codebooks with adaptive mixing weights	(Liu et al., 2023)
Speech recognition	Accent-specific codebooks per accent group	(Prabhu et al., 2023)
Model quantization/compression	Codebooks per neuron/probe/parameter group	(Yvinec et al., 2023)
CAD sequence generation	Disentangled codebooks for topology/geometry/extrusion	(Xu et al., 2022)
MIMO precoding	Product codebooks for dimensionally-structured channel	(Bhogi et al., 2021)
Semi-quantitative group testing	Codeword grouping via λ-parameterized adder-maps	(Chen et al., 2024)

In deep generative architectures, grouping enhances codebook utilization, allows for modular specialization (e.g., per image region, sound accent, or object class), and mitigates codeword collapse. In MIMO, product codebooks match tensor decompositions of the channel and enable high-rate, memory-efficient precoding selection (Bhogi et al., 2021). In sketch/extrude CAD, disentangled codebooks capture independently variable factors with fine-grained user controllability (Xu et al., 2022).

4. Adaptive, Disentangled, and Modular Codebook Designs

Grouped codebooks are key to modern adaptive and disentangled representations:

• Disentanglement: Multi-codebook designs permit model components to specialize to independent, semantically meaningful factors (e.g., topology, geometry, extrusion in CAD (Xu et al., 2022)), enabling conditional generation, code-mixing, and precise control over the generative or reconstruction process.
• Adaptivity: AdaCode (Liu et al., 2023) constructs local, adaptive codebooks as weighted combinations of basis codebooks, allowing the network to allocate capacity dynamically across spatial locations and image content, outperforming fixed single-codebook baselines.
• Specialization: Accent-specific codebooks in ASR (Prabhu et al., 2023) allow specialization and improved recognition performance via cross-attention, especially under domain shift (e.g., unseen accents).
• Post-hoc Resizing: Group-VQ (Zheng et al., 15 Oct 2025) supports training-free resampling and extension of codebooks per group, affording flexible adaptation of overall code capacity without retraining.

These architectures are especially advantageous when heterogeneous or structured variation in the data is irreducible to a single global codebook.

5. Theoretical Bounds and Statistical Guarantees

Grouped codebook constructions are associated with explicit bounds on achievable rates, codebook sizes, and redundancy, governed by manifold geometry, statistical estimation, and information-theoretic principles:

• In semi-quantitative group testing, λ-ADD codes achieve rates $$C = \bigcup_{g=1}^G G_g, \quad G_g \cap G_{g'} = \emptyset \text{ for } g \neq g'$$0 for constant $$C = \bigcup_{g=1}^G G_g, \quad G_g \cap G_{g'} = \emptyset \text{ for } g \neq g'$$1, and possess Gilbert-Varshamov-type lower bounds in the proportional-distance regime (Chen et al., 2024).
• In high-dimensional MIMO quantization, performance criteria are stated in terms of minimizing average mutual information loss under quantization, mapped to chordal distances on product Grassmann manifolds (Bhogi et al., 2021).
• In neural codebook quantization, codebook utilization and reconstruction quality are jointly optimized via group-wise losses, with post-training metrics (rFID, PSNR) quantifying model fidelity (Zheng et al., 15 Oct 2025).

Empirically, optimal group size and codebook cardinality for maximal utilization and accuracy are found via ablation and are application-dependent.

6. Limitations, Tradeoffs, and Practical Considerations

Grouped codebooks introduce both flexibility and complexity. Key considerations include:

• Over/under-utilization: Too many groups can resemble vanilla VQ, risking codebook collapse; too few can wash out expressivity by excessive parameter sharing (Zheng et al., 15 Oct 2025).
• Memory vs. expressivity: Increasing the number of groups and intra-group codewords improves representation capacity but increases storage. Balanced codebook partitioning is crucial.
• Assignment overhead: While grouping may avoid explicit mapping overhead (e.g., via row-range index encoding), adaptive or input-dependent codebook selection can add computational cost.
• Optimization stability: Unguided group assignments can lead to utilization imbalance; entropy regularization, weight penalties, or custom gradient updates are used to stabilize training (Liu et al., 2023, Yvinec et al., 2023).
• Compatibility with architecture: Some grouping schemes require pre-processing (e.g., clustering) or architectural changes (e.g., cross-attention, per-group training), which may be non-trivial to integrate into legacy systems.

7. Representative Results and Impact

Grouped codebook designs yield statistically significant gains across domains:

• Neural image compression and VQ-VAEs: Group-VQ achieves rFID=1.86, LPIPS=0.11 with utilization ≈100% on ImageNet 128×128, outperforming joint and vanilla VQ (Zheng et al., 15 Oct 2025).
• Image restoration: AdaCode elevates PSNR for 4× super-resolution to 27.80 (vs. 27.45 fixed codebook), improves perceptual and FID metrics, and reduces observed artifacts (Liu et al., 2023).
• Accent-robust ASR: Accent-specific codebooks reduce WER for seen accents by up to 36.3% and for unseen accents by up to 11.6% relative in the MCV-ACCENT-100 benchmark (Prabhu et al., 2023).
• Quantized model deployment: JLCM compresses LLaMA-7B to 2 GB with only minor accuracy drop (95% of original) and outperforms all prior quantization methods at similar ratios (Yvinec et al., 2023).
• Generative CAD models: Disentangled codebooks in SkexGen produce FID=18.56 for sketch generation (vs. 75.47 DeepCAD), with coverage, novelty, and diversity improvements (Xu et al., 2022).
• FD-MIMO codebooks: Product codebook learning on the Cartesian Product Grassmann manifold yields memory and search speedup with negligible information loss (Bhogi et al., 2021).

These results verify that grouped codebooks, whether implemented by explicit partitioning, product structure, or context-adaptive mixture, provide both efficiency and high-fidelity modeling across a broad spectrum of computational tasks.