---
title: Effective Codebook Capacity
url: https://www.emergentmind.com/topics/effective-codebook-capacity
type: topic
---

# Effective Codebook Capacity

Effective codebook capacity is the usable informational or representational budget of a codebook, as distinct from its nominal cardinality. Recent arXiv literature formalizes the notion in domain-specific ways: as the cumulative entropic budget of position-dependent codebooks in autoregressive visual generation, bounded by the dataset information content [2605.06207]; as entropy or utilization of empirical code-selection distributions in vector quantization and residual vector quantization [2410.12359]; and as realized mutual information or achievable rate in communication and mismatched-decoding settings [2508.04223; 2206.10123]. Across these formulations, the common principle is that raw codebook size does not by itself determine what can be learned, transmitted, or effectively used.

## 1. Domain-dependent definitions

The phrase “effective codebook capacity” does not denote a single universal scalar. Instead, it is instantiated according to the operative constraint in each domain: finite dataset entropy, codebook-collapse dynamics, compositional expressivity, or channel-induced uncertainty.

| Setting | Formalization | Limiting factor |
|---|---|---|
| AR visual generation | \(I(t)=\sum_{i=1}^{t}\log_2 K_i\) | \(\log_2 N\) dataset bits |
| RVQ audio codecs | \(C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)\) | Collapse, imbalance, redundancy |
| VQ tokenizers | \(K\times U\) or \(H(Z)\) | Sparse activation |
| Semantic communication | \(I(Z;Y)\) or \(I(S;\hat S)\) | Channel noise, input law |
| Mismatched decoding | Achievable rate \(I(P_X;W)\) | Decoder metric |

In the autoregressive image setting, the central object is not \(K^L\) or \(K\times L\), but the cumulative per-position entropic budget \(I(L)=\sum_{t=1}^{L}\log_2 K_t\), with the further restriction that no more than \(\log_2 N\) bits in the training set can ever be “filled” [2605.06207]. In RVQ, ERVQ defines theoretical capacity as \(C_{\mathrm{th}}=M\cdot \log_2 K\) and effective capacity as the entropy sum over empirical codeword-selection distributions, making bit-efficiency \(BE=C_{\mathrm{eff}}/C_{\mathrm{th}}\) the natural normalized quantity [2410.12359]. In VQ-based image tokenizers, effective capacity is sometimes treated more operationally as the number of distinct active codes, \(K\times U\), or as the entropy \(H(Z)\) of the discrete latent distribution [2509.10140; 2406.11837].

This suggests that the term is best understood as a family resemblance rather than a single invariant. What unifies the usage is an insistence on realized structure—entropy, active support, mutual information, or achievable rate—rather than nominal symbol inventory.

## 2. Information-theoretic formulation in autoregressive visual generation

In “Taming the Entropy Cliff,” images are encoded into sequences \(\{x_1,\dots,x_L\}\), with \(x_t\) drawn from a codebook of size \(K\). Two bounds are immediate: the dataset contains at most
\[
E = H(x_1,\dots,x_L)\le \log_2 N,
\]
and each position satisfies
\[
H(x_t\mid x_{<t})\le \log_2 K.
\]
For a uniform codebook, after \(t\) positions the model can consume at most \(t\log_2 K\) bits, so equating this with the dataset limit yields the entropy-cliff boundary
\[
t^*=\left\lceil \frac{\log_2 N}{\log_2 K}\right\rceil.
\]
Beyond \(t^*\), the training-set conditional distribution \(p(x_t\mid x_{<t})\) collapses to a point mass, and the AR model cannot learn a meaningful probability distribution; it must memorize continuations [2605.06207].

The paper emphasizes that naive capacity surrogates such as \(K\times L\) and \(K^L\) are misleading in finite-data regimes. A constant codebook with large \(K\) may have enormous nominal combinatorial size while exhausting the learnable information budget almost immediately. On ImageNet, with \(N\approx 1.28\times 10^6\) and \(K=16384\), one has \(\log_2 N\approx 20.3\) bits and \(\log_2 K=14\) bits, giving \(t^*\approx 2\). The reported measured conditional entropy drops to zero after two tokens, so the remaining 254 positions in a 256-token sequence become a memorization problem [2605.06207].

A notable contrast is drawn with language. The same phenomenon is “not observed in language,” because its natural structure keeps the effective entropy per position well below the codebook capacity. That contrast is important because it locates the problem not in autoregression per se, but in the interaction among dataset entropy, symbol allocation, and sequence order.

## 3. Variable allocation of capacity and the entropy cliff

The proposed remedy is Variable Codebook Size Quantization (VCQ), which replaces the constant \(K\) with a monotone schedule \(\{K_t\}_{t=1}^{L}\) growing from \(K_{\min}\) to \(K_{\max}\):
\[
K_t = K_{\min} + (K_{\max}-K_{\min})\,f\!\Bigl(\frac{t-1}{L-1}\Bigr),
\qquad f(0)=0,\; f(1)=1.
\]
The paper considers three convex-on-\([0,1]\) schedules: Linear, Cosine, and Power with \(f(\tau)=\tau^\alpha\), \(\alpha>1\). The effective codebook capacity in bits is then
\[
I(t)=\sum_{i=1}^{t}\log_2 K_i.
\]
The key claim is that capacity must be distributed along the sequence in a way that matches the decaying conditional entropy of the data, not merely maximized in aggregate [2605.06207].

Implementation preserves the standard training pipeline. All VQ-GAN losses remain unchanged, including reconstruction \(L1\), LPIPS, PatchGAN adversary, and VQ commitment. The AR transformer uses the usual next-token cross-entropy objective. A single shared codebook of size \(K_{\max}\) is maintained, and at position \(t\) nearest-neighbor search is restricted to the first \(K_t\) entries. No extra parameters or losses are introduced, and the optimizer, learning-rate schedule, and weight decay are identical to the uniform-codebook baseline [2605.06207].

On ImageNet, the paper uses \(K_{\min}=2\), \(K_{\max}=16384\), and a Cosine schedule, which pushes the empirical entropy cliff from \(t^*=2\) to \(t^*\approx 5\). Reported generation and probing results are summarized below.

| Configuration | Metric | Result |
|---|---|---|
| Uniform \(K=16384\) | gFID w/o CFG | 27.98 |
| VCQ Cosine | gFID w/o CFG | 14.80 |
| Uniform \(K=16384\) | gFID with CFG | 6.43 |
| VCQ Cosine | gFID with CFG | 4.79 |
| VCQ scaled up | gFID | 1.71 |
| Uniform, first 10 tokens | Top-1 linear probe | 27.1% |
| VCQ Cosine, first 10 tokens | Top-1 linear probe | 43.8% |

The scaled model reaches gFID \(1.71\) with \(684\)M autoregressive parameters, “without any extra training techniques such as semantic regularization or causal alignment” [2605.06207]. The severe \(1\)-bit bottleneck at \(t=1\) induces what the paper describes as a natural coarse-to-fine semantic hierarchy: first-10-token linear probing improves from \(27.1\%\) under a uniform codebook to \(43.8\%\) under VCQ-Cosine, while all-256-token accuracy is \(48.0\%\) for the uniform baseline and \(48.8\%\) for VCQ-Cosine; VCQ-Power with \(\alpha=2.5\) gives \(49.4\%\) for the first 10 tokens and \(46.9\%\) for all 256 tokens [2605.06207].

A common misconception is that increasing \(K\) is sufficient. The VCQ results directly challenge that view: the paper’s conclusion is that what matters is not only how many bits are allocated, but where they are allocated along the autoregressive sequence.

## 4. Utilization, collapse, and compositional expansion in vector quantization

Outside autoregressive image generation, effective codebook capacity is often tied to code usage rather than sequence position. In ERVQ, codebook collapse means that only \(K_m\ll K\) entries are ever selected in a codebook \(m\); if those active entries are equally used, then \(H(p^m)=\log_2 K_m < \log_2 K\), and the effective capacity drop per codebook is \(\Delta C_m=\log_2(K/K_m)\). ERVQ addresses this with online clustering, a code-balancing loss, and an inter-codebook SSIM penalty. On APCodec with \(M=4\) and \(K=1024\), the reported utilization rates move from \([14.7\%,16.3\%,25.5\%,41.2\%]\) to \([100\%,100\%,100\%,100\%]\), while bit-efficiency rises from \(0.766\) to \(0.976\) [2410.12359].

Several later VQ papers make the same diagnostic move with different mechanisms. CVQ-VAE defines effective codebook capacity as the number of codevectors actually used over the dataset and combats collapse by re-seeding under-used codes toward anchors sampled from the current feature distribution; on CIFAR10 with a 1024-entry codebook, the paper reports Usage \(=100\%\), Perplexity \(\approx 1024\), and rFID \(=24.7\) [2307.15139]. Chang et al.’s VQBridge explicitly treat effective capacity as \(K\times U\), where \(U\) is utilization rate, and report \(100\%\) usage even at \(K=262{,}144\), with ImageNet \(256\times256\) reconstruction rFID improving from \(1.30\) at \(16\)k and 40 epochs to \(0.95\) at \(262\)k and 40 epochs, and to \(0.88\) at \(262\)k and 120 epochs [2509.10140]. VQGAN-LC instead freezes a \(100{,}000\)-entry codebook derived from pre-trained vision features, trains a projector, and defines effective capacity as \(H(Z)\), bounded by \(\log_2 N\); at \(N=100\)k it reports utilization \(99.9\%\) and rFID \(2.62\) on ImageNet [2406.11837].

A different line of work expands effective capacity compositionally. LooC splits a \(d\)-dimensional feature vector into \(m\) blocks of dimension \(d^*=d/m\), quantizes each block using a shared low-dimensional codebook, and thereby expands the representational space from \(K\) to \(K^m\) while storing only \(K\cdot d/m\) parameters. The paper reports \(100\%\) utilization on every dataset and codebook size tested, and on CIFAR-10 compares CVQ-VAE \(1024\times 128\) against LooC \(256\times 4\) with \(m=32\): rFID improves from \(24.73\) to \(19.22\) and PSNR from \(24.72\) to \(34.51\) [2601.00222]. Dual Codebook VQ uses parallel global and local codebooks with joint capacity \(C_{\mathrm{eff}}=N_g\times N_l\), so a balanced split gives \(N^2/4\); the paper reports utilization rising from approximately \(20\%\) in vanilla VQ-GAN to nearly \(100\%\) in each half, alongside MS-COCO FID \(4.19\) with a total codebook of \(512\) [2503.10832].

These papers show that, in VQ systems, “effective capacity” typically means either entropy of actual selections or the active support of the codebook, sometimes augmented by compositional structure. The recurring pathology is not lack of nominal symbols, but inability to activate or diversify them.

## 5. Communication-theoretic and beamforming interpretations

In communication settings, effective codebook capacity is typically formulated as mutual information or achievable rate. In the mismatched-decoding analysis of Merhav and Böcherer, the encoder uses a constant-composition subcode but the decoder searches over the full linear code with an additive metric. The main result is that codebook mismatch can be fully compensated: the optimal mismatched metric achieves the constant-composition random-coding exponent, and the maximal achievable rate under mismatch is
\[
\max\{R:\exists\,U,\;E_r(R,U)>0\}=I(P_X;W).
\]
As \(R\to I(P_X;W)\), the optimal metric converges to the MAP metric, so the effective capacity under codebook mismatch is exactly the mutual information for the chosen input assignment [2206.10123].

Digital semantic communication papers adopt closely related definitions. One formulation treats \(S=q(Z)\) as a deterministic quantization of a continuous semantic feature \(Z\), yielding \(I(Z;S)=H(S)\le \log K\), with equality under uniform activation \(\pi_k=1/K\); the paper then introduces an entropy-regularized quantization loss and a channel-aware semantic distortion loss, and reports PSNR improvement \(24.1\%\) and LPIPS improvement \(46.5\%\) at SNR \(=10\) dB relative to existing codebook designs [2510.07108]. Another formulation defines effective capacity as \(I(S;\hat S)\) under learned activation probabilities and AWGN, with a Wasserstein regularizer that aligns the empirical activation law to a hybrid of uniform and Gaussian targets; WS-DC is reported to improve inference accuracy while keeping model size at approximately \(7.4\)M parameters and inference time unchanged [2508.04223]. A related channel-aware discrete semantic coding framework likewise aligns the marginal code-activation distribution with the channel-optimal input distribution \(\pi^*(z)\) by penalizing the \(1\)-Wasserstein distance \(W_1(\pi_\phi,\pi^*)\), and ties effective codebook capacity to \(I(Z;Y)=H(Z)-H(Z\mid Y)\) [2508.04291].

Beamforming literature uses the term operationally, in terms of how many steering vectors are truly needed to realize near-maximal array gain or satisfy a capacity constraint. One mmWave study derives a closed-form beam coverage condition under a gain-loss tolerance \(\gamma\) dB and shows that even a codebook as large as \(1024^{16}\) can be reduced to a small set of steering vectors: for \(\gamma=2\) dB, only \(M=9\) beams are needed for a \(4\)-element ULA and \(M=17\) for a \(4\times 4\) URA, while over \(90\%\) of random trials stay within the \(\gamma\) dB bound [2505.09162]. By contrast, the beam-squint compensation work imposes a minimum per-beam channel-capacity constraint and shows that beam squint reduces channel capacity and requires denser codebooks; for a \(64\)-antenna ULA at \(73\) GHz with \(2.5\) GHz bandwidth, the proposed algorithm improves channel capacity by \(17.8\%\) and analysis suggests that squint limits the feasible bandwidth for a given array size [1705.04441].

Here the same nominal pattern reappears in a different guise: large codebooks may be unnecessary when beam coverage is the operative bottleneck, but insufficiently dense codebooks become harmful when wideband beam squint imposes a channel-capacity constraint.

## 6. Conceptual synthesis and recurring misconceptions

A first misconception is that nominal cardinality is an adequate proxy for representational power. The surveyed literature repeatedly rejects this. In autoregressive image modeling, \(K^L\) and \(K\times L\) do not capture what a finite dataset allows the model to learn [2605.06207]. In VQ tokenizers, large \(K\) with low utilization yields low effective capacity, often with severe collapse [2509.10140]. In beamforming, even astronomically large steering codebooks can be reducible to \(O(10)\) useful beams under an explicit gain-loss tolerance [2505.09162].

A second misconception is that maximizing codebook usage is always sufficient. The entropy-cliff analysis shows a more subtle failure mode: a uniform codebook can have abundant nominal availability and still allocate too much capacity too early, exhausting the \(\log_2 N\) budget after only a few positions [2605.06207]. Conversely, communication papers show that non-uniform activation can be desirable when the target is not usage uniformity but proximity to the capacity-achieving input law under a noisy channel [2508.04291; 2508.04223].

A third misconception is that compact codebooks necessarily imply reduced effective capacity. LooC increases representational space from \(K\) to \(K^m\) while shrinking parameter count to \(K\cdot d/m\) [2601.00222]. Dual Codebook VQ obtains joint capacity \(N_gN_l\) from two smaller books [2503.10832]. VCQ improves generation quality without altering loss, parameter count, or AR training procedure, by reorganizing where capacity is spent rather than enlarging the total budget [2605.06207].

Taken together, these results support a broad technical reading of effective codebook capacity as the realizable information budget after accounting for data entropy, codebook allocation along sequences, empirical usage statistics, combinatorial composition, decoder mismatch, and channel constraints. The unifying lesson is not merely that “bigger is not always better,” but that usable capacity is governed by the interaction between codebook structure and the process—learning, compression, or transmission—that must exploit it.

Source: https://www.emergentmind.com/topics/effective-codebook-capacity