Codebook Collapse: Causes & Mitigations

Updated 4 July 2026

Codebook collapse is a phenomenon where LLM-guided political event coding fails to preserve expert mapping and vector quantization underuses its discrete latent space.
In LLM settings, enriched prompting and detailed operational probes (e.g., macro-F1, Rule-S) help assess and mitigate the loss of coding logic.
In vector quantization, strategies like deferred quantization and global codebook adaptations improve code utilization and enhance representational capacity.

Codebook collapse denotes distinct failure modes in two research traditions. In LLM-guided political event coding, it is the failure of a model to preserve the coding logic of an expert codebook under controlled perturbations, even when clearer codebooks substantially improve predictive accuracy (He et al., 4 Jun 2026). In vector quantization, VQ-VAE, VQGAN, latent diffusion, neural audio codecs, and related tokenizers, it is the degeneration of a discrete codebook such that only a small subset of entries is effectively selected and updated, while many codes become dead or severely underused (Zheng et al., 2023, Zhao et al., 2024). The shared term therefore names different pathologies: in one case, a failure of semantic rule preservation; in the other, a failure of discrete representational utilization.

1. Terminological scope and domain-specific meanings

The phrase codebook collapse is not semantically uniform across the literature. In social-science measurement with LLMs, the “codebook” is an expert-written ontology and rule set for mapping text to structured labels. In discrete representation learning, the “codebook” is a learned dictionary of vectors or tokens used by a quantizer. The term is consequently polysemous rather than merely broad.

Research area	Meaning of codebook collapse	Primary diagnostics
LLM political event coding	Failure to preserve coding logic under codebook-preserving perturbations	Macro-F1, accuracy, CB-Align., Rule-S, Fleiss’ $\kappa$ , Generic-label F1, Swap-mapping F1
Vector quantization	Only a small subset of code vectors or tokens is selected and updated	Utilization, active-code ratio, perplexity, usage histograms, MSE, rFID, LPIPS

Within VQ research, further terminological refinement has emerged. Zhao, Zou, Shah, and Liu distinguish tokens collapse from embeddings collapse: the former is concentration of nearest-neighbor assignments on a small subset of tokens, while the latter is upstream degeneration of encoder outputs before quantization (Zhao et al., 2024). Other papers retain the older usage and call the low-utilization phenomenon codebook collapse (Zheng et al., 2023, Zhu et al., 2024). In speech tokenization, the same pathology is often called index collapse (Guo et al., 2024).

2. Codebook collapse in LLM-guided political event coding

In the political event coding literature, codebook collapse arises in a task that is explicitly described as “beyond ordinary sentence-level classification” (He et al., 4 Jun 2026). Political event coding converts text into structured source–action–target records. The studied task is relation classification: given a text and a designated source–target pair, the model predicts the directed political action the source took, or did not take, toward the target. The ontology is PLOVER, built on the CAMEO framework, with a three-level hierarchy: Binary (Cooperation vs. Conflict), Quad (Verbal vs. Material Cooperation or Conflict), and Root, which contains 15 fine-grained categories including AGREE, CONSULT, SUPPORT, COOPERATE, AID, YIELD, REQUEST, ACCUSE, REJECT, THREATEN, PROTEST, SANCTION, MOBILIZE, COERCE, and ASSAULT (He et al., 4 Jun 2026).

Event-mode rules are central. Similar surface wording can map to different labels depending on whether the action is threatened or promised, completed, or halted or negated. For the source–target pair Protesters $\rightarrow$ Government, “said they would stage demonstrations” maps to THREATEN, “staged demonstrations” maps to PROTEST, and “ended demonstrations” maps to YIELD (He et al., 4 Jun 2026). The failure mode called codebook collapse is therefore not simple class confusion; it concerns whether the model actually follows the supplied operational distinctions.

The paper studies several forms of codebook operationalization. A No Codebook baseline gives the text, marked source–target spans, and the valid label set. Compact prompting adds short label-specific natural-language definitions, quad-level groupings, and global disambiguation. Enriched prompting keeps the same label inventory but adds worked examples per label, event-mode guidance, and boundary notes for commonly confused categories such as future cooperation versus completed aid, halted aid, CONSULT overuse, peacekeeping nuances, and Material Conflict priority (He et al., 4 Jun 2026). The study also evaluates in-context learning, chain-of-thought scaffolds, retrieval-augmented generation with FAISS indexes and Sentence-BERT embeddings, a hierarchical RAG variant, and a structured JSON representation of the codebook; the latter decreased performance in unconstrained prompting relative to Compact natural language (He et al., 4 Jun 2026).

The central distinction in this literature is between predictive accuracy and behavioral reliability (He et al., 4 Jun 2026). Predictive accuracy is the share of correct labels under the original unperturbed codebook, measured as macro-F1 or accuracy. Behavioral reliability is invariance of the coding function’s outputs when the codebook is altered in surface form while preserving the intended operational definitions. The probes hold the text and source–target pair fixed, and vary only the codebook presentation through legal-label compliance checks, definition recovery, order perturbations, generic-label perturbations, and swapped-mapping perturbations. The methodology is formalized as a codebook $C=(L,D,R)$ , text $x$ , coding function $f(x;C)\in L$ , transformation $T$ , and alignment permutation $\pi$ , with behavioral reliability defined as

$B = P_{x,T}\big[f(x;C)=\pi(f(x;T(C)))\big].$

Two operational diagnostics summarize the probes. Codebook Alignment (CB-Align.) is the average of valid-label compliance and definition-recovery accuracy. Rule-following Score (Rule-S) is the average of order $\kappa$ , Generic-label F1, and Swap-mapping F1. These are reported explicitly as operational diagnostics rather than separate theoretical constructs (He et al., 4 Jun 2026).

Empirically, clearer codebooks improve predictive performance but not behavioral reliability. On PLV root-level evaluation, the mean macro-F1 across Gemma2:9B, Qwen2.5:7B, Mistral-7B-Instruct, and Llama-3.1-8B-Instruct rises from 0.457 with No Codebook to 0.574 with Compact and 0.633 with Enriched. On AW binary evaluation, the mean macro-F1 rises from 0.757 to 0.782 to 0.794 (He et al., 4 Jun 2026). Yet on PLV, CB-Align. increases from 0.967 to 0.990 while Rule-S slightly declines from 0.471 to 0.461. The probe breakdown is even more revealing: order $\kappa$ improves from 0.743 to 0.782, but Generic-label F1 falls from 0.566 to 0.519 and Swap-mapping F1 falls from 0.105 to 0.081. On AW, CB-Align. is nearly perfect, but Swap-mapping F1 remains low at 0.141 for Compact and 0.109 for Enriched (He et al., 4 Jun 2026).

The error analyses show why the authors use the phrase codebook collapse. In a swapped-mapping probe, “Merkel met Erdogan for talks on a migration deal” has gold CONSULT and the model predicts CONSULT under the original mapping; after CONSULT’s definition is reassigned and the expected output becomes COOPERATE, the model still outputs CONSULT. Likewise, “North Korea threatened to shell the Christmas tree” still yields THREATEN when THREATEN’s definition is reassigned to AID (He et al., 4 Jun 2026). The model recognizes valid labels and can recover definitions, yet it fails to preserve the supplied label–definition association. This undermines measurement validity, comparability across coding schemes, and robustness to codebook revisions in downstream social-science analyses.

4. Codebook collapse in vector quantization and discrete tokenization

In VQ systems, codebook collapse is the classical failure mode in which only a small subset of codevectors receives useful gradients, while the majority “dies off” and remains near random initialization (Zheng et al., 2023). The standard setting has an encoder $\rightarrow$ 0, decoder $\rightarrow$ 1, and codebook $\rightarrow$ 2. Continuous latents $\rightarrow$ 3 are quantized by nearest neighbor,

$\rightarrow$ 4

and the canonical VQ-VAE loss is

$\rightarrow$ 5

EMA or k-means-like updates are frequently used for selected codes, but repeated non-selection leaves dead codes without gradient or EMA updates (Zheng et al., 2023).

The standard diagnostics are active-code fraction, occupancy histograms, entropy, and perplexity. For assignment probabilities $\rightarrow$ 6, codebook perplexity is

$\rightarrow$ 7

and low perplexity signals concentration of assignments on a few codes (Zheng et al., 2023). Zhao, Zou, Shah, and Liu further distinguish tokens collapse, where nearest-neighbor assignments concentrate around a few modes, from embeddings collapse, where insufficient encoder capacity causes encoder outputs for different classes or modes to merge before quantization (Zhao et al., 2024). “Early Quantization Shrinks Codebook” adds another geometric perspective: even when dead-token statistics appear less extreme, the average pairwise Euclidean distance among codes can shrink sharply, indicating a narrow and clustered codebook geometry (Zhao et al., 17 Mar 2026).

The phenomenon is documented across image, audio, and speech tokenizers. In CVQ-VAE, plain VQ-VAE on CIFAR10 uses only 9.96% of codes, whereas CVQ-VAE reaches 100% usage; in VQGAN, usage is 42% on FFHQ and 44% on ImageNet, while CVQ-VAE again reaches 100% (Zheng et al., 2023). In VQGAN with $\rightarrow$ 8 and $\rightarrow$ 9, vanilla utilization is approximately 1.4% and VQGAN-EMA reaches approximately 4.5%, while SimVQ reaches approximately 100% utilization across codebooks as large as 262k (Zhu et al., 2024). In large-codebook speech tokenizers, single-codebook VQ-VAE collapses severely at 65,536 codes, whereas product-quantized VAE maintains high usage and much larger perplexity (Guo et al., 2024). In RVQ-based audio codecs, APCodec baseline utilization can fall to 14.7%, 16.3%, 25.5%, and 41.2% across four layers, while ERVQ reaches 100% in all four codebooks (Zheng et al., 2024).

5. Mechanistic explanations across VQ variants

Different papers isolate different proximate causes, but the recurring theme is that winner-take-all training creates a self-reinforcing mismatch between encoder geometry and codebook evolution. One explanation is disjoint optimization: only the selected code vector receives gradient, so selected entries move toward the encoder manifold while non-selected entries remain frozen. SimVQ makes this argument explicit and attributes collapse to nearest-neighbor assignment plus straight-through training, which updates only the chosen row of the codebook matrix (Zhu et al., 2024). A related systems-level account is encoder drift: as the encoder moves the latent distribution, sparsely updated codes lag behind, lose assignments, and thereby drift even further out of contention. NSVQ formalizes this feedback loop as encoder drift $C=(L,D,R)$ 0 codebook lag $C=(L,D,R)$ 1 dead codes $C=(L,D,R)$ 2 larger quantization residuals $C=(L,D,R)$ 3 a larger straight-through gradient-estimation gap, and treats nonstationarity rather than static imbalance as the central cause (Lu et al., 9 Jun 2026). “Beyond Stationarity” makes the same point in stochastic-approximation terms and proposes that nonstationary encoder updates, rather than only bad initialization or large $C=(L,D,R)$ 4, are fundamental to collapse (Lu et al., 21 Feb 2026).

A second family of accounts emphasizes initialization and latent geometry. Zhao, Zou, Shah, and Liu show that K-means or EMA initialization on an untrained encoder’s outputs yields centroids concentrated in a narrow region because the untrained encoder already compresses distinct data modes into only a few latent peaks; reducing encoder hidden size further induces embeddings collapse (Zhao et al., 2024). “Early Quantization Shrinks Codebook” sharpens this argument by showing that enabling quantization too early pushes both encoder outputs and codebook entries into a narrow latent region, reducing pairwise code distances and later token diversity. On ImageNet-100, for example, deferred quantization yields much larger codebook Euclidean distance and much higher perplexity than early quantization for both MaskGIT and VAR tokenizers (Zhao et al., 17 Mar 2026).

Other subliteratures identify more specialized mechanisms. In discrete VAEs, EdVAE argues that softmax produces overconfident probability distributions over code indices, repeatedly selecting a small subset of codes and starving the rest; its remedy is evidential uncertainty via a Dirichlet layer rather than direct softmax (Baykal et al., 2023). In smoothed vector quantization, code collapse appears when simplex assignments remain far from one-hot while still concentrating on a few vertices; the proposed regularizer explicitly pushes assignments toward all simplex vertices via a per-vertex $C=(L,D,R)$ 5-nearest-neighbor objective (Morita, 26 Sep 2025). In visual tokenizers for complex scenes, SGC-VQGAN ties collapse to codebooks learned only from pixel reconstruction, producing semantically inconsistent and imbalanced tokens; semantic online clustering and consistent semantic learning are then used to align codes with segmentation-derived class structure (Ding et al., 2024). In RVQ audio codecs, collapse is compounded by inter-layer redundancy: early codebooks absorb most information, later layers quantize weak residuals, and adjacent layers become too similar, which motivates both intra-codebook balancing and inter-codebook diversity penalties in ERVQ (Zheng et al., 2024).

6. Mitigation strategies, architectural alternatives, and broader significance

The anti-collapse literature is heterogeneous because the diagnosed causes differ. One large class of methods preserves the VQ formalism but improves code maintenance. CVQ-VAE treats reinitialization as an online clustering problem: it tracks decayed usage $C=(L,D,R)$ 6, selects anchors from current encoder features, and moves low-used codes toward anchors with usage-dependent step sizes, achieving 100% usage across VQ-VAE, VQGAN, and latent diffusion while improving reconstruction and generation (Zheng et al., 2023). NSVQ adds a dense non-stationary embedding loss, codebook replacement in early training, and stage-wise encoder freezing; on ImageNet-1k at $C=(L,D,R)$ 7 with 65,536 codes, it improves rFID from 2.39 to 2.10 relative to SimVQ while both retain 100% utilization (Lu et al., 9 Jun 2026). TransVQ similarly attempts global codebook adaptation through a lightweight mapping while preserving convergence to the k-means solution (Lu et al., 21 Feb 2026). FVQ or VQBridge uses a compress–process–recover projector plus learning-rate annealing to address straight-through bias, one-step-behind updates, and sparse codebook gradients, achieving 100% usage even with a 262k codebook and rFID 0.88 after 120 epochs on ImageNet $C=(L,D,R)$ 8 (Chang et al., 12 Sep 2025).

A second class changes the parameterization of the quantizer. SimVQ freezes a raw codebook and learns a single linear map $C=(L,D,R)$ 9 so that the entire codebook subspace moves jointly; this yields near-complete utilization up to 262k codes in image and audio settings (Zhu et al., 2024). LooC replaces a monolithic $x$ 0 codebook with a shared low-dimensional compositional codebook applied across $x$ 1 subspaces, expanding effective combinations from $x$ 2 to $x$ 3 while shrinking parameters from $x$ 4 to $x$ 5; it reports 100% usage and strong reconstruction with much smaller codebooks (Li et al., 1 Jan 2026). PQ-VAE factorizes speech features into multiple VQ subspaces and adds dual decoding from continuous and quantized paths, which improves per-subspace perplexity and avoids index collapse at large effective codebook sizes (Guo et al., 2024). ERVQ augments RVQ with online clustering, a code balancing loss, and inter-codebook SSIM minimization, reaching 100% utilization in APCodec and improving ViSQOL, STOI, and LSD across several codecs (Zheng et al., 2024).

A third class changes training schedules or priors rather than the discrete operator alone. Deferred Quantization trains a continuous autoencoder first and introduces quantization only after the encoder has learned a dispersed latent geometry, thereby counteracting early shrinkage and improving both perplexity and codebook spread (Zhao et al., 17 Mar 2026). VQCT injects pretrained language-model codebook priors and part-of-speech structure into vector-quantized image modeling, generating adjective and noun codebooks through a graph convolutional transfer network rather than learning unrelated code vectors from scratch (Zhang et al., 2024). SGC-VQGAN adds segmentation-derived semantic embeddings and multi-level clustering, reporting 100% active tokens together with strong semantic uniqueness and clustering metrics on complex scenes (Ding et al., 2024). EdVAE introduces a Dirichlet-Categorical posterior that regularizes uncertainty and mitigates overconfident discrete posteriors, increasing perplexity and improving reconstruction and generation on CIFAR10, CelebA, and LSUN Church (Baykal et al., 2023). In smoothed VQ, the simplex-vertex $x$ 6NN regularizer provides a direct route to both one-hotness and balanced usage (Morita, 26 Sep 2025).

The most radical response is to remove the codebook altogether. PCA-VAE replaces VQ with an online PCA bottleneck trained by Oja’s rule, eliminating codebooks, commitment losses, and lookup noise. Because there is no discrete nearest-neighbor codebook, the specific failure mode called codebook collapse does not arise; the paper reports stronger reconstruction than VQ-GAN and SimVQ on CelebA-HQ while using 10–100 $x$ 7 fewer latent bits (Lu et al., 21 Feb 2026).

Across both major meanings of the term, codebook collapse is fundamentally a fidelity problem. In LLM political event coding, the issue is whether outputs remain tied to the expert coding logic that makes social-science variables meaningful (He et al., 4 Jun 2026). In vector quantization, the issue is whether a nominally large discrete latent space is actually used as a high-capacity representation rather than collapsing to a small active subset (Zheng et al., 2023). The literatures therefore converge on a common methodological lesson: neither codebook size nor nominal codebook quality is sufficient. What matters is whether model behavior remains stably anchored to the codebook’s intended operational role.