Dual-Codebook Architecture

• Dual-codebook architecture is a design that uses two separate codebooks to capture complementary aspects of latent representations, enhancing expressivity and reconstruction quality.
• It leverages parallel, disjoint, and hierarchical quantization strategies to achieve robust performance in applications such as image VQ, speech tokenization, and 3D point cloud modeling.
• Empirical studies demonstrate over 95% codebook utilization and significant gains in reconstruction fidelity and error probability reduction compared to single-codebook systems.

A dual-codebook architecture is any model design in which two separate codebooks—collections of representative discrete vectors—operate jointly, either in parallel or in distinct roles, to improve compactness, expressivity, or diversity of representation. Dual-codebook mechanisms have arisen in lossy source-channel coding, vector quantized representation learning, speech tokenization, model compression, 3D point cloud modeling, and generative recommendation. This entry surveys state-of-the-art formulations, objectives, coding-theoretic implications, training methodologies, and quantitative outcomes across these domains.

1. Dual-Codebook Principles and Formulations

Dual-codebook architectures partition latent representations so that distinct codebooks capture complementary aspects of the input space. Typical partitioning strategies include:

• Splitting the latent vector into disjoint subspaces, each quantized by a different codebook (as in product VQ (Guo et al., 2024) and dual VQ (Malidarreh et al., 13 Mar 2025)).
• Assigning codebooks to different representational levels (e.g., shallow vs. deep features in point cloud completion (Wu et al., 19 Jan 2025)).
• Routing based on item characteristics, such as popularity versus semantic content in recommendation (Hui et al., 15 Nov 2025).
• Allocating disjoint codebooks to distinct decoders or user groups to leverage diversity in a joint source-channel coding context (Rowan et al., 15 Jan 2026).

Mathematically, for an input feature $x \in \mathbb{R}^d$, dual-codebook quantization computes

$z = Q_1(x_1) + Q_2(x_2)$

or, in concatenative product quantization,

$z = \text{concat}(Q_1(x_1), Q_2(x_2)),$

where $x_1, x_2$ are partitions of $x$ and $Q_1, Q_2$ index separate codebooks, possibly of differing size, structure, or update dynamics.

Duality may also arise in the grouping of model parameters and the assignment of group-specific codebooks for quantization, as in memory footprint compression (Yvinec et al., 2023), or in codebook allocation controlled by a learnable router (Hui et al., 15 Nov 2025).

2. Dual-Codebook Architectures in Source-Channel Coding

In broadcast joint source-channel coding (JSCC), the dual-codebook paradigm fundamentally alters the diversity achieved at the system level (Rowan et al., 15 Jan 2026):

• Channel diversity (single shared codebook): Each decoder receives a different channel output but reconstructs from the same codebook.
• Codebook diversity (disjoint/dual codebooks): Each decoder has its own subcodebook, resulting in $K$ independent trials to recover a good match.

The hybrid architecture partitions $K$ decoders into $J$ groups, assigning a shared subcodebook to each group, thus interpolating between full codebook and channel diversity. This is formally realized by partitioning codeword indices via a marked Poisson point process and optimizing success probability via first- and second-order achievability bounds.

For the disjoint-codebook scheme, the one-shot ensemble error probability is tightly bounded: $$P_e \leq \mathbb{E}_{W,X,Y}\left[\left(1 + K\,P_Z(\mathcal{B}_D(W))\,2^{\iota_{X;Y}(X;Y)}\right)^{-1}\right],$$ whereas the hybrid approach replaces $K$ with $J$ and maximizes over codebook groups. Performance on the BSC demonstrates that for large $K$, codebook diversity outperforms shared-codebook/channel diversity, while carefully tuning $J$ yields strictly better performance than either extreme (Rowan et al., 15 Jan 2026).

3. Vector Quantization and Dual-Codebook Learning

Modern vector quantization frameworks exploit dual-codebook strategies to enhance utilization and reconstructive power.

In Dual Codebook VQ (Malidarreh et al., 13 Mar 2025):

• Two codebooks of size $K$ operate in parallel: (i) a global codebook $\mathcal{C}_g$ updated via a lightweight Transformer, and (ii) a local codebook $\mathcal{C}_l$ updated via deterministic nearest-neighbor assignment.
• Latent features $x$ are split, quantized separately, and merged—either summed or concatenated—before decoding.
• The overall loss is a VQ-GAN hybrid, comprising reconstruction, codebook, and commitment losses, as well as adaptive GAN balancing.

Empirical results show that dual-codebook VQ achieves $>95\%$ utilization for both codebooks, avoiding collapse, and surpasses strong single-book baselines even at half the codebook size.

In point cloud completion (Wu et al., 19 Jan 2025), the encoder codebook $C_E$ captures regional geometric patterns at shallow feature levels, while the decoder codebook $C_D$ quantizes fine-grained deep features. An explicit Quantized Information Exchange (QIE) mechanism (code deduplication, re-targeting via MLP, and code merging) aligns and fuses these representations, reducing variability of surface coverage and ambiguity inherent to high-dimensional sampling.

Dual-codebook architectures also appear in product-quantized VAEs (PQ-VAE) for speech tokenization (Guo et al., 2024), where the latent space is partitioned and separately quantized, with both continuous and quantized decoders sharing supervision. This approach increases codebook perplexity, usage, and robustness to index collapse, scaling to implicit codebook sizes exceeding $2^{16}$ with strong reconstruction fidelity.

4. Adaptive and Task-Conditioned Dual-Codebooks

Duality in codebook structure can reflect semantic priors or application-specific needs. In generative recommendation (FlexCode (Hui et al., 15 Nov 2025)), two codebooks—one for collaborative-filtering signals and another for semantic content—distribute a fixed token budget adaptively per item. A Mixture-of-Experts (MoE) router, parametrized by item popularity and sparsity, determines the allocation: $$L_{\text{cf}}(i) = \lfloor \alpha_i \cdot B \rfloor, \qquad L_{\text{sem}}(i) = B - L_{\text{cf}}(i)$$ where $\alpha_i$ is the CF allocation ratio. Alignment and smoothness terms in the joint loss drive the two codebooks to coherent, smoothly-varying representations over the item popularity spectrum.

Empirically, FlexCode yields superior NDCG and tail performance over single codebook and fixed-split baselines, with particularly robust gains for long-tail items and under tight token budgets.

5. Codebook Assignment, Optimization, and Proximal Gradients

Dual- and multi-codebook designs require principled assignment and learning of both codebook vectors and mapping indices. In DNN compression (Yvinec et al., 2023), JLCM groups neurons by local distributional similarity (clustering), applies separate codebooks per group (dual or more), and learns both codewords and hard assignment maps jointly.

The learning objective

$$\mathcal{L}(C,I) = \|\tilde f_{C,I}(\tilde X) - f_{\rm fp16}(X)\|^2 + \|C \cdot \rm{softmax}(I) - W\|^2 + \lambda \sum_{i,j}(1 - |2 \, \rm{softmax}(I_{i,j})-1|^{\beta})$$

incorporates activation mimicking, quantization error, and a “harden-softmax” penalty to push mappings toward discrete indices. Critically, a custom proximal operator for the mapping gradients prioritizes minimal-distance codeword transitions, counteracting the tendency of SGD to “jump” to far-off extreme centroids. The result is higher accuracy and memory compression—e.g., retaining $\sim 95\%$ accuracy in Llama 7B models at massive storage reduction.

6. Quantitative Outcomes and Comparative Performance

Empirical studies in recent dual-codebook architectures demonstrate:

Domain	Dual-Codebook Role	Key Metrics/Findings	Reference
Broadcast JSCC	Decoders/access groups	Achievability bounds show hybrid strictly outperforms single diversity; hybrid parameter $J$ tunable for optimal error	(Rowan et al., 15 Jan 2026)
Image VQ	Global/local partition; sum/concat	$>95\%$ codebook utilization; FID improvement on ADE20K, MS-COCO, CelebA-HQ; surpasses single-book larger models	(Malidarreh et al., 13 Mar 2025)
Speech PQ-VAE	Chunking encoder, dual decoding	Avoids codebook collapse; perplexity increased; reconstruction RMSE reduced	(Guo et al., 2024)
Point Cloud	Shallow/deep codebook + QIE	Reduces ambiguity due to sampling; state-of-the-art on PCN/ShapeNet; K≈512, R≈128 optimal	(Wu et al., 19 Jan 2025)
Recommendation	Adaptive router (popularity/gating)	NDCG/HR improved on both head and tail; ablations confirm necessity of duality and dynamic allocation	(Hui et al., 15 Nov 2025)
Compression	Parameter-group/vectors + scales	7–8× size reduction, ≈2–3 bpp, 1.2–2 pt ImageNet top-1 boost vs. single-codebook GPTQ at same bit budget	(Yvinec et al., 2023)

Successful dual-codebook designs consistently display:

• Higher codebook utilization (reduced collapse).
• More efficient capacity allocation (e.g., smaller codebooks attaining better reconstruction under GAN or MSE losses).
• Improved robustness to distributional or data imbalance (e.g., popularity in recommendations, surface coverage in 3D, channel uncertainty in JSCC).

7. Theoretical Considerations and Variants

Key theoretical implications of dual-codebook designs include:

• Diversity trade-offs: In JSCC, tuning the codebook partition parameter balances codebook and channel diversity, leading to strictly improved non-asymptotic bounds (Rowan et al., 15 Jan 2026).
• Avoidance of index collapse: Product quantization with dual or multi codebooks ensures that each codeword continues to receive gradients, even for large implicit codebooks (Guo et al., 2024).
• Explicit regularization: Cross-codebook alignment and smoothing (in generative recsys (Hui et al., 15 Nov 2025)), and contrastive or commitment losses (in VQ/image or 3D models (Malidarreh et al., 13 Mar 2025, Wu et al., 19 Jan 2025)), maintain coherence and efficient coverage of the representational space.
• Modularity and scaling: Factorizing large codebooks into dual or multiple smaller tables—possibly with routing, scaling, or deduplicating mechanisms—enables models to scale to large vocabularies, low memory targets, or highly imbalanced data domains.

Plausibly, further variants will arise as more domains identify disentangled, complementary representational axes suitable for codebook splitting, and as alignment methodologies become more sophisticated. The modular nature of dual-codebook architectures also facilitates their integration into multi-task, multi-modal, or cross-domain systems.

---

For a comprehensive review of these advances, see (Rowan et al., 15 Jan 2026, Malidarreh et al., 13 Mar 2025, Wu et al., 19 Jan 2025, Hui et al., 15 Nov 2025, Yvinec et al., 2023), and (Guo et al., 2024).