Effective Codebook Capacity

• Effective Codebook Capacity is the realized informational budget determined by entropy, active support, and mutual information rather than the nominal size of the codebook.
• It underpins methods in autoregressive visual generation, vector quantization, and semantic communication by guiding how bits are allocated along a sequence or across models.
• Empirical studies, including VCQ approaches, show that strategic capacity allocation improves image generation quality and representation efficiency without increasing model complexity.

Effective codebook capacity is the usable informational or representational budget of a codebook, as distinct from its nominal cardinality. Recent arXiv literature formalizes the notion in domain-specific ways: as the cumulative entropic budget of position-dependent codebooks in autoregressive visual generation, bounded by the dataset information content (Zheng et al., 7 May 2026); as entropy or utilization of empirical code-selection distributions in vector quantization and residual vector quantization (Zheng et al., 2024); and as realized mutual information or achievable rate in communication and mismatched-decoding settings (Zhang et al., 6 Aug 2025, Merhav et al., 2022). Across these formulations, the common principle is that raw codebook size does not by itself determine what can be learned, transmitted, or effectively used.

1. Domain-dependent definitions

The phrase “effective codebook capacity” does not denote a single universal scalar. Instead, it is instantiated according to the operative constraint in each domain: finite dataset entropy, codebook-collapse dynamics, compositional expressivity, or channel-induced uncertainty.

Setting	Formalization	Limiting factor
AR visual generation	$I(t)=\sum_{i=1}^{t}\log_2 K_i$	$\log_2 N$ dataset bits
RVQ audio codecs	$C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$	Collapse, imbalance, redundancy
VQ tokenizers	$K\times U$ or $H(Z)$	Sparse activation
Semantic communication	$I(Z;Y)$ or $I(S;\hat S)$	Channel noise, input law
Mismatched decoding	Achievable rate $I(P_X;W)$	Decoder metric

In the autoregressive image setting, the central object is not $K^L$ or $K\times L$, but the cumulative per-position entropic budget $\log_2 N$0, with the further restriction that no more than $\log_2 N$1 bits in the training set can ever be “filled” (Zheng et al., 7 May 2026). In RVQ, ERVQ defines theoretical capacity as $\log_2 N$2 and effective capacity as the entropy sum over empirical codeword-selection distributions, making bit-efficiency $\log_2 N$3 the natural normalized quantity (Zheng et al., 2024). In VQ-based image tokenizers, effective capacity is sometimes treated more operationally as the number of distinct active codes, $\log_2 N$4, or as the entropy $\log_2 N$5 of the discrete latent distribution (Chang et al., 12 Sep 2025, Zhu et al., 2024).

This suggests that the term is best understood as a family resemblance rather than a single invariant. What unifies the usage is an insistence on realized structure—entropy, active support, mutual information, or achievable rate—rather than nominal symbol inventory.

2. Information-theoretic formulation in autoregressive visual generation

In “Taming the Entropy Cliff,” images are encoded into sequences $\log_2 N$6, with $\log_2 N$7 drawn from a codebook of size $\log_2 N$8. Two bounds are immediate: the dataset contains at most

$\log_2 N$9

and each position satisfies

$C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$0

For a uniform codebook, after $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$1 positions the model can consume at most $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$2 bits, so equating this with the dataset limit yields the entropy-cliff boundary

$C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$3

Beyond $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$4, the training-set conditional distribution $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$5 collapses to a point mass, and the AR model cannot learn a meaningful probability distribution; it must memorize continuations (Zheng et al., 7 May 2026).

The paper emphasizes that naive capacity surrogates such as $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$6 and $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$7 are misleading in finite-data regimes. A constant codebook with large $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$8 may have enormous nominal combinatorial size while exhausting the learnable information budget almost immediately. On ImageNet, with $C_{\mathrm{eff}}=\sum_{m=1}^{M} H(p^m)$9 and $K\times U$0, one has $K\times U$1 bits and $K\times U$2 bits, giving $K\times U$3. The reported measured conditional entropy drops to zero after two tokens, so the remaining 254 positions in a 256-token sequence become a memorization problem (Zheng et al., 7 May 2026).

A notable contrast is drawn with language. The same phenomenon is “not observed in language,” because its natural structure keeps the effective entropy per position well below the codebook capacity. That contrast is important because it locates the problem not in autoregression per se, but in the interaction among dataset entropy, symbol allocation, and sequence order.

3. Variable allocation of capacity and the entropy cliff

The proposed remedy is Variable Codebook Size Quantization (VCQ), which replaces the constant $K\times U$4 with a monotone schedule $K\times U$5 growing from $K\times U$6 to $K\times U$7: $K\times U$8 The paper considers three convex-on-$K\times U$9 schedules: Linear, Cosine, and Power with $H(Z)$0, $H(Z)$1. The effective codebook capacity in bits is then

$H(Z)$2

The key claim is that capacity must be distributed along the sequence in a way that matches the decaying conditional entropy of the data, not merely maximized in aggregate (Zheng et al., 7 May 2026).

Implementation preserves the standard training pipeline. All VQ-GAN losses remain unchanged, including reconstruction $H(Z)$3, LPIPS, PatchGAN adversary, and VQ commitment. The AR transformer uses the usual next-token cross-entropy objective. A single shared codebook of size $H(Z)$4 is maintained, and at position $H(Z)$5 nearest-neighbor search is restricted to the first $H(Z)$6 entries. No extra parameters or losses are introduced, and the optimizer, learning-rate schedule, and weight decay are identical to the uniform-codebook baseline (Zheng et al., 7 May 2026).

On ImageNet, the paper uses $H(Z)$7, $H(Z)$8, and a Cosine schedule, which pushes the empirical entropy cliff from $H(Z)$9 to $I(Z;Y)$0. Reported generation and probing results are summarized below.

Configuration	Metric	Result
Uniform $I(Z;Y)$1	gFID w/o CFG	27.98
VCQ Cosine	gFID w/o CFG	14.80
Uniform $I(Z;Y)$2	gFID with CFG	6.43
VCQ Cosine	gFID with CFG	4.79
VCQ scaled up	gFID	1.71
Uniform, first 10 tokens	Top-1 linear probe	27.1%
VCQ Cosine, first 10 tokens	Top-1 linear probe	43.8%

The scaled model reaches gFID $I(Z;Y)$3 with $I(Z;Y)$4M autoregressive parameters, “without any extra training techniques such as semantic regularization or causal alignment” (Zheng et al., 7 May 2026). The severe $I(Z;Y)$5-bit bottleneck at $I(Z;Y)$6 induces what the paper describes as a natural coarse-to-fine semantic hierarchy: first-10-token linear probing improves from $I(Z;Y)$7 under a uniform codebook to $I(Z;Y)$8 under VCQ-Cosine, while all-256-token accuracy is $I(Z;Y)$9 for the uniform baseline and $I(S;\hat S)$0 for VCQ-Cosine; VCQ-Power with $I(S;\hat S)$1 gives $I(S;\hat S)$2 for the first 10 tokens and $I(S;\hat S)$3 for all 256 tokens (Zheng et al., 7 May 2026).

A common misconception is that increasing $I(S;\hat S)$4 is sufficient. The VCQ results directly challenge that view: the paper’s conclusion is that what matters is not only how many bits are allocated, but where they are allocated along the autoregressive sequence.

4. Utilization, collapse, and compositional expansion in vector quantization

Outside autoregressive image generation, effective codebook capacity is often tied to code usage rather than sequence position. In ERVQ, codebook collapse means that only $I(S;\hat S)$5 entries are ever selected in a codebook $I(S;\hat S)$6; if those active entries are equally used, then $I(S;\hat S)$7, and the effective capacity drop per codebook is $I(S;\hat S)$8. ERVQ addresses this with online clustering, a code-balancing loss, and an inter-codebook SSIM penalty. On APCodec with $I(S;\hat S)$9 and $I(P_X;W)$0, the reported utilization rates move from $I(P_X;W)$1 to $I(P_X;W)$2, while bit-efficiency rises from $I(P_X;W)$3 to $I(P_X;W)$4 (Zheng et al., 2024).

Several later VQ papers make the same diagnostic move with different mechanisms. CVQ-VAE defines effective codebook capacity as the number of codevectors actually used over the dataset and combats collapse by re-seeding under-used codes toward anchors sampled from the current feature distribution; on CIFAR10 with a 1024-entry codebook, the paper reports Usage $I(P_X;W)$5, Perplexity $I(P_X;W)$6, and rFID $I(P_X;W)$7 (Zheng et al., 2023). Chang et al.’s VQBridge explicitly treat effective capacity as $I(P_X;W)$8, where $I(P_X;W)$9 is utilization rate, and report $K^L$0 usage even at $K^L$1, with ImageNet $K^L$2 reconstruction rFID improving from $K^L$3 at $K^L$4k and 40 epochs to $K^L$5 at $K^L$6k and 40 epochs, and to $K^L$7 at $K^L$8k and 120 epochs (Chang et al., 12 Sep 2025). VQGAN-LC instead freezes a $K^L$9-entry codebook derived from pre-trained vision features, trains a projector, and defines effective capacity as $K\times L$0, bounded by $K\times L$1; at $K\times L$2k it reports utilization $K\times L$3 and rFID $K\times L$4 on ImageNet (Zhu et al., 2024).

A different line of work expands effective capacity compositionally. LooC splits a $K\times L$5-dimensional feature vector into $K\times L$6 blocks of dimension $K\times L$7, quantizes each block using a shared low-dimensional codebook, and thereby expands the representational space from $K\times L$8 to $K\times L$9 while storing only $\log_2 N$00 parameters. The paper reports $\log_2 N$01 utilization on every dataset and codebook size tested, and on CIFAR-10 compares CVQ-VAE $\log_2 N$02 against LooC $\log_2 N$03 with $\log_2 N$04: rFID improves from $\log_2 N$05 to $\log_2 N$06 and PSNR from $\log_2 N$07 to $\log_2 N$08 (Li et al., 1 Jan 2026). Dual Codebook VQ uses parallel global and local codebooks with joint capacity $\log_2 N$09, so a balanced split gives $\log_2 N$10; the paper reports utilization rising from approximately $\log_2 N$11 in vanilla VQ-GAN to nearly $\log_2 N$12 in each half, alongside MS-COCO FID $\log_2 N$13 with a total codebook of $\log_2 N$14 (Malidarreh et al., 13 Mar 2025).

These papers show that, in VQ systems, “effective capacity” typically means either entropy of actual selections or the active support of the codebook, sometimes augmented by compositional structure. The recurring pathology is not lack of nominal symbols, but inability to activate or diversify them.

5. Communication-theoretic and beamforming interpretations

In communication settings, effective codebook capacity is typically formulated as mutual information or achievable rate. In the mismatched-decoding analysis of Merhav and Böcherer, the encoder uses a constant-composition subcode but the decoder searches over the full linear code with an additive metric. The main result is that codebook mismatch can be fully compensated: the optimal mismatched metric achieves the constant-composition random-coding exponent, and the maximal achievable rate under mismatch is

$\log_2 N$15

As $\log_2 N$16, the optimal metric converges to the MAP metric, so the effective capacity under codebook mismatch is exactly the mutual information for the chosen input assignment (Merhav et al., 2022).

Digital semantic communication papers adopt closely related definitions. One formulation treats $\log_2 N$17 as a deterministic quantization of a continuous semantic feature $\log_2 N$18, yielding $\log_2 N$19, with equality under uniform activation $\log_2 N$20; the paper then introduces an entropy-regularized quantization loss and a channel-aware semantic distortion loss, and reports PSNR improvement $\log_2 N$21 and LPIPS improvement $\log_2 N$22 at SNR $\log_2 N$23 dB relative to existing codebook designs (Wang et al., 8 Oct 2025). Another formulation defines effective capacity as $\log_2 N$24 under learned activation probabilities and AWGN, with a Wasserstein regularizer that aligns the empirical activation law to a hybrid of uniform and Gaussian targets; WS-DC is reported to improve inference accuracy while keeping model size at approximately $\log_2 N$25M parameters and inference time unchanged (Zhang et al., 6 Aug 2025). A related channel-aware discrete semantic coding framework likewise aligns the marginal code-activation distribution with the channel-optimal input distribution $\log_2 N$26 by penalizing the $\log_2 N$27-Wasserstein distance $\log_2 N$28, and ties effective codebook capacity to $\log_2 N$29 (Zhang et al., 6 Aug 2025).

Beamforming literature uses the term operationally, in terms of how many steering vectors are truly needed to realize near-maximal array gain or satisfy a capacity constraint. One mmWave study derives a closed-form beam coverage condition under a gain-loss tolerance $\log_2 N$30 dB and shows that even a codebook as large as $\log_2 N$31 can be reduced to a small set of steering vectors: for $\log_2 N$32 dB, only $\log_2 N$33 beams are needed for a $\log_2 N$34-element ULA and $\log_2 N$35 for a $\log_2 N$36 URA, while over $\log_2 N$37 of random trials stay within the $\log_2 N$38 dB bound (Bozkurt et al., 14 May 2025). By contrast, the beam-squint compensation work imposes a minimum per-beam channel-capacity constraint and shows that beam squint reduces channel capacity and requires denser codebooks; for a $\log_2 N$39-antenna ULA at $\log_2 N$40 GHz with $\log_2 N$41 GHz bandwidth, the proposed algorithm improves channel capacity by $\log_2 N$42 and analysis suggests that squint limits the feasible bandwidth for a given array size (Cai et al., 2017).

Here the same nominal pattern reappears in a different guise: large codebooks may be unnecessary when beam coverage is the operative bottleneck, but insufficiently dense codebooks become harmful when wideband beam squint imposes a channel-capacity constraint.

6. Conceptual synthesis and recurring misconceptions

A first misconception is that nominal cardinality is an adequate proxy for representational power. The surveyed literature repeatedly rejects this. In autoregressive image modeling, $\log_2 N$43 and $\log_2 N$44 do not capture what a finite dataset allows the model to learn (Zheng et al., 7 May 2026). In VQ tokenizers, large $\log_2 N$45 with low utilization yields low effective capacity, often with severe collapse (Chang et al., 12 Sep 2025). In beamforming, even astronomically large steering codebooks can be reducible to $\log_2 N$46 useful beams under an explicit gain-loss tolerance (Bozkurt et al., 14 May 2025).

A second misconception is that maximizing codebook usage is always sufficient. The entropy-cliff analysis shows a more subtle failure mode: a uniform codebook can have abundant nominal availability and still allocate too much capacity too early, exhausting the $\log_2 N$47 budget after only a few positions (Zheng et al., 7 May 2026). Conversely, communication papers show that non-uniform activation can be desirable when the target is not usage uniformity but proximity to the capacity-achieving input law under a noisy channel (Zhang et al., 6 Aug 2025, Zhang et al., 6 Aug 2025).

A third misconception is that compact codebooks necessarily imply reduced effective capacity. LooC increases representational space from $\log_2 N$48 to $\log_2 N$49 while shrinking parameter count to $\log_2 N$50 (Li et al., 1 Jan 2026). Dual Codebook VQ obtains joint capacity $\log_2 N$51 from two smaller books (Malidarreh et al., 13 Mar 2025). VCQ improves generation quality without altering loss, parameter count, or AR training procedure, by reorganizing where capacity is spent rather than enlarging the total budget (Zheng et al., 7 May 2026).

Taken together, these results support a broad technical reading of effective codebook capacity as the realizable information budget after accounting for data entropy, codebook allocation along sequences, empirical usage statistics, combinatorial composition, decoder mismatch, and channel constraints. The unifying lesson is not merely that “bigger is not always better,” but that usable capacity is governed by the interaction between codebook structure and the process—learning, compression, or transmission—that must exploit it.