Hierarchical & Codebook-Based Representations

• Hierarchical and codebook-based representations are structured methods that discretize high-dimensional data through multi-level quantization and symbolic abstraction to achieve compactness and scalability.
• They employ multi-stage quantizers where coarse codes capture global structure and finer codes refine local details, ensuring efficient search and robust reconstruction.
• These approaches drive practical applications in generative modeling, multimodal tokenization, and wireless beamforming by significantly reducing computational load while maintaining high fidelity.

Hierarchical and codebook-based representations are structured approaches for mapping high-dimensional data or signal spaces onto discrete sets of symbols, enabling efficient data compression, generative modeling, search, and downstream inference. Such representations are foundational to modern deep learning in generative modeling, multimodal learning, wireless communications, and large-scale text and vision tasks. At their core, hierarchical and codebook-based approaches exploit multi-level quantization or symbolic abstraction, leading to compactness, compositionality, and computational scalability.

1. Foundational Principles and Motivation

Hierarchical and codebook-based representations extend classical vector quantization by introducing multiple codebooks or multi-stage quantizers. Rather than relying on a single, flat look-up table, hierarchical methods organize the code space such that a coarse code (or set of codes) identifies a broad region in the feature manifold, and subsequent finer codes express residual or local detail within that region. This recursive partitioning enables exponential representational capacity with linearly scaling search/decoding complexity and improved robustness to codebook collapse or underutilization.

Codebook-based approaches are not restricted to Euclidean vector spaces. They are also defined on Riemannian manifolds (e.g., hyperbolic spaces for trees), multi-modal representations (where alignment and disentanglement across codebooks is central), and domain-specific combinatorial structures (e.g., tree codes for text). This adaptability underpins their utility across a broad spectrum of applications, including discrete latent variable models, beamforming/CSI search in wireless communications, neural codecs for images/audio, and efficient semantic tokenization for text or multimodal content.

2. Architectures: Hierarchies and Interleaved Codebooks

A prototypical hierarchical codebook architecture consists of an encoder mapping input $x$ to continuous latents, a sequence of codebooks at different abstraction levels, and a decoder synthesizing outputs from the discrete codes. Representative instantiations include:

• HR-VQVAE: Each layer quantizes the residual left by the previous layer via a vector-quantized encoder. The codebook structure is a tree, with $n$ levels and codebook size $m$ per node. At spatial location $(h, w)$, only the codebook of the parent codeword is searched per layer, so encoding cost is $O(nm)$, not exponential in tree depth. The final representation is the sum of codewords across all layers, supporting efficient decoding and generative modeling (Adiban et al., 2022).
• Hyperbolic Residual Quantization (HRQ): The quantization is performed in the Poincaré ball, with Möbius operations replacing Euclidean addition/subtraction. At each level $\ell$, the codeword is drawn from a codebook defined on $\mathbb{P}^n_c$, with residual computation based on Möbius subtraction. This framework naturally exploits the exponential expansion of hyperbolic geometry to match branching hierarchies (Piękos et al., 18 May 2025).
• Multi-modal and Semantic Hierarchies: In semantic-guided architectures such as SemHiTok, a pretrained semantic codebook (coarse) is used to route each token to a pixel/texture sub-codebook (fine), with hierarchical decoupling of codebook learning stages (semantic first, pixel second). In text-aligned VQ (TA-VQ), multiple codebook levels align with different semantic text granularities (word, phrase, sentence), each level coupled via optimal transport or contrastive objectives to matching textual representations (Chen et al., 9 Mar 2025, Liang et al., 3 Mar 2025).
• Beamforming and Sensing Codebooks: Massive MIMO and RIS systems employ binary-tree codebooks over spatial, angular, or near-field domains. Each layer refines the angular or spatial sector by subdividing the parent sector, with child codewords designed (via e.g., Gerchberg–Saxton synthesis, AMCF optimization, or manifold alternation) to match hardware, propagation, or blockage constraints (Zhang et al., 14 Nov 2025, Shi et al., 2022, Qi et al., 2022, Carlson et al., 2023).

3. Training Objectives, Optimization, and Collapse Prevention

Training typically targets both fidelity of the final reconstruction/generation and robust utilization of the entire codebook hierarchy.

• Hierarchical VQ Loss: Each layer uses a per-layer loss combining (i) an MSE term to the input (or residual), (ii) a codebook update term (pushing codewords toward inputs), and (iii) a commitment loss (pulling encoder outputs toward selected codewords). HR-VQVAE's global loss includes a top-level tie between the sum of codewords and the encoder output. There is no ELBO/KL term; stochasticity is introduced only in extensions like HQ-VAE (Adiban et al., 2022, Takida et al., 2023).
• Bayesian Stochastic Quantization: HQ-VAE generalizes VQ-VAE to stochastic code selection via variational Bayes, learning per-layer quantization noise parameters $s_l^2$ (self-annealing), and modeling hierarchies via ladder (injected or residual) structures. Gumbel-softmax is used for discrete code sampling (Takida et al., 2023).
• Alignment and Information Objectives: In semantic or text-aligned models, hierarchical codebooks are jointly or sequentially trained to match features at each semantic level with suitable textual embeddings, with optimal-transport, Sinkhorn, or contrastive losses enforcing cross-modal code–text alignment. Cross-modal information disentangling is realized via mutual information regularization/minimization between coarse/fine layers and between modality-specific/general features (Liang et al., 3 Mar 2025, Huang et al., 2024).
• Collapse Prevention: Using hierarchical codebook decomposition (multiple small codebooks per layer, conditionally activated), as in HR-VQVAE, stabilizes utilization and avoids catastrophic "dead codewords" typical in flat VQ codebooks with large cardinalities. Bayesian or stochastic quantization further ensures diversity via annealing; codebook collapse is less likely as latent partitioning is localized (Adiban et al., 2022, Takida et al., 2023).

4. Efficient Decoding, Scalability, and Practical Search

Hierarchical structure confers major efficiency advantages:

Feature	Flat Codebook	Hierarchical Codebook
Codebook Size	$M$	$n \cdot m$ (for $n$0 layers, size $n$1/layer)
Encoding/Decoding Complexity	$n$2	$n$3
Scalability (to large codebooks)	Poor (collapse, slow)	Good (linear scaling, stable as $n$4)
Search per Query	O(M) VQ	Tree or sequential: $n$5; beam search $n$6
Codebook Utilization	Collapses for large $n$7	Stable; each small codebook covers local residual manifold

Empirically, HR-VQVAE reconstructs 10,000 FFHQ 256$n$8256 images in 0.84s, versus 5.1s (VQVAE) and 9.3s (VQVAE-2). In MIMO and RIS, hierarchical codebooks cut probe complexity from $n$9 to $m$0, with negligible rate loss at realistic SNRs (Adiban et al., 2022, Zhang et al., 14 Nov 2025, Shi et al., 2022, Qi et al., 2022).

5. Application Domains and Empirical Outcomes

Hierarchical and codebook-based representations are deployed across:

• Neural Compression and Generation: Hierarchical VQ-VAEs (HR-VQVAE, HQ-VAE, RQ-VAE, HRQ) are state-of-the-art in image and audio modeling. Hierarchies provide lower reconstruction errors and improved FID across datasets; ablations consistently show depth and codebook size yield monotonic quality gain (if collapse is averted) (Adiban et al., 2022, Takida et al., 2023, Piękos et al., 18 May 2025).
• Multimodal Tokenization and Understanding: SemHiTok’s two-level decoupled semantic–pixel codebook produces joint tokens effective for both multimodal understanding (LLaVA-1.5) and generation (GenEval, MJHQ30K), achieving state-of-the-art discrete MLLM performance. Hierarchical TA-VQ closes the gap between code and language structure, boosting fine-grained semantic alignment and downstream VQA/captioning (Chen et al., 9 Mar 2025, Liang et al., 3 Mar 2025).
• Wireless Beamforming and Sensing: Hierarchical codebooks in mmWave/THz MIMO, RIS, DMAs, and conformal arrays drive near-optimal beam selection with orders-of-magnitude lower training/feedback overhead. Application-specific enhancements (blockage awareness, spatial chirp, subarray patterning) further adapt the hierarchy to hardware and propagation constraints, with hierarchical schemes achieving 92–99% of exhaustive performance at 1–10% of search cost (Zhang et al., 14 Nov 2025, Carlson et al., 2023, Shi et al., 2022, Wei et al., 2021, Qi et al., 2022, Zhang et al., 2020).
• Efficient Symbolic Recognition: HierCode’s binary-tree hierarchical codebook for Chinese text leverages linguistic structural priors, enabling compact parameterization and robust zero-shot (OOV) performance. Shared radical prototypes in the hierarchy allow recomposition for unseen characters, attaining recognition accuracy on par with or better than flat one-hot classifiers with a 92.6% reduction in classification layer parameters (Zhang et al., 2024).

6. Unified Probabilistic and Algorithmic Frameworks

Classical codebook design—especially for visual words and compact histogram representations—is systematized by hierarchical merging under explicit probabilistic models. “A Generalized Probabilistic Framework” unifies a variety of merging criteria:

• Model/Estimation Grid:

Model (class-conditional)	Estimation	Merging Score / Method
Multinomial	Maximum Likelihood	Agglomerative IB (AIB)
Multinomial	Bayesian	MLT (Dirichlet-smoothed)
Multinomial	Max Margin	MME (linear SVM criterion)
Gaussian	Maximum Likelihood	GMLE
Gaussian	Bayesian	UVD
Gaussian	Scatter Matrix	CSM

Across classification/detection tasks, max-margin–informed multinomial hierarchies (MME) yield most compact yet discriminative codebooks. Bayesian and MLE variants address overfitting and sample bias; Gaussian models adapt better to dense (non-sparse) descriptors (Liu et al., 2014).

7. Future Directions and Open Challenges

Key open directions in hierarchical and codebook-based representations include:

• Scaling to ultra-high dimensions and modalities: Integrating hierarchical codebook architectures into very-high-dimensional latent spaces and across diverse modalities (e.g., text, images, audio, structured knowledge graphs) while maintaining codebook stability and efficiency.
• Hyperbolic, Riemannian, and domain-specific manifolds: Further generalizing codebooks via Riemannian geometry—HRQ demonstrates substantial gains for hierarchical data; open challenges remain in scaling to vision and non-tree domains (Piękos et al., 18 May 2025).
• End-to-end cross-domain and semantic alignment: Jointly training codebook hierarchies for cross-modal retrieval, generation, and understanding in complex scenarios (multi-granular, multi-lingual, task-adaptive settings).
• Algorithmic advances for inference and search: Improving hierarchical search algorithms to handle adversarial examples, hardware constraints, and real-time feedback in dynamic environments without performance degradation.

This area remains central to scalable, efficient, and interpretable deep learning and signal processing, bridging the gap between discrete symbolic reasoning and high-capacity continuous modeling.