Latent Codebooks for Fast Thinking

• Latent Codebooks for Fast Thinking are methods that replace explicit chain-of-thought with compact latent representations to preserve strategy while reducing token cost.
• They leverage learned strategy prototypes and attention mechanisms to inject concise thinking tokens within transformer layers, accelerating inference.
• Adaptive routing and control mechanisms, such as GainRouter, balance fast latent reasoning with explicit deliberation for improved performance across diverse tasks.

Searching arXiv for the cited papers to ground the article in current literature. Latent codebooks for fast thinking are mechanisms that replace, compress, or control explicit Chain-of-Thought (CoT) with internal latent representations so that a model can preserve strategy-level guidance while reducing autoregressive reasoning length, latency, and token cost. In the most direct formulation, a model learns a finite bank of reasoning prototypes from concise CoT sketches and, at inference, conditions on a small set of latent “thinking vectors” rather than generating a long rationale; related work broadens the idea to steering directions, latent reasoning extractors, short chains of latent thoughts, and reserved abstract reasoning vocabularies (Zheng et al., 28 Sep 2025, Lin et al., 4 Jul 2025, Ramji et al., 24 Apr 2026).

1. Conceptual scope and terminology

The immediate motivation is the inefficiency of explicit reasoning traces. Reasoning-oriented LLMs often rely on CoT, self-consistency, or tree-of-thought style decoding, but long reasoning traces increase wall-clock time and token usage, and can also induce “overthinking,” where extra steps introduce irrelevant detours and surface-level heuristics that degrade accuracy. In the LC-FT study, Qwen3-Thinking uses approximately $19$k tokens on AIME, which is presented as a concrete example of this cost regime (Zheng et al., 28 Sep 2025).

Within this literature, “fast thinking” denotes inference in which the model produces the answer directly, guided by a compact internal representation of reasoning strategy rather than by explicit textual steps. “Slow thinking” denotes explicit multi-step generation in token space. The distinction appears both in models that learn reusable strategy priors and in models that dynamically modulate reasoning speed at test time (Zheng et al., 28 Sep 2025, Lin et al., 4 Jul 2025).

The term “codebook” is used in more than one sense. In LC-FT, the codebook is a learnable matrix of discrete strategy prototypes, indexed discretely but accessed through soft attention, yielding continuous mixtures (Zheng et al., 28 Sep 2025). In “Controlling Thinking Speed in Reasoning Models,” a single layerwise steering vector $v^l$ functions as a one-dimensional latent code for the slow-fast transition, and the paper explicitly motivates extending that single axis into a set of reasoning-mode directions (Lin et al., 4 Jul 2025). In “Abstract Chain-of-Thought,” the codebook is a reserved abstract vocabulary $\mathcal{V}_{\mathrm{abs}}$ of discrete non-linguistic tokens generated under constrained decoding (Ramji et al., 24 Apr 2026). In “CoLT,” by contrast, there is no discrete VQ-style codebook; the latent units are continuous hidden states placed at designated reasoning positions (Hu et al., 30 Jun 2026).

This suggests a useful working distinction. A narrow usage refers to finite prototype banks or reserved vocabularies. A broader usage refers to any compact latent intermediate representation that mediates reasoning and can be controlled, selected, or routed more efficiently than full verbal CoT.

2. Latent Codebooks for Fast Thinking in LLMs

LC-FT defines the codebook as

$\mathbf{C} \in \mathbb{R}^{M \times H},$

where each row $c_j \in \mathbb{R}^H$ is a strategy prototype. A small, fixed set of learnable queries

$\mathbf{Q} \in \mathbb{R}^{K \times H}$

retrieves mixtures of these prototypes by attention, with $K$ much smaller than the number of tokens in a CoT trace, and the paper gives $16$–$48$ as representative values. The resulting latent hints are the thinking tokens

$$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$

where $v^l$0 is the attention matrix over codebook entries (Zheng et al., 28 Sep 2025).

The training signal does not use full verbose rationales. Instead, the method constructs concise CoT sketches $v^l$1. A teacher model proposes a candidate hint, and the hint is accepted only if it does not leak the answer and if the backbone model, when prompted with the question plus hint, produces an answer that a task-specific verifier accepts. The resulting dataset is

$v^l$2

These sketches are used only during training, not at inference (Zheng et al., 28 Sep 2025).

The thinking tokens are injected at an internal transformer layer $v^l$3. If the text input has hidden states $v^l$4 at layer $v^l$5, LC-FT appends the latent slots: $v^l$6 The injected vectors are not input tokens; they are non-textual latent slots. They are masked before layer $v^l$7, so earlier layers process only the original token sequence, and from layer $v^l$8 onward the model attends jointly over text states and thinking tokens. A small refiner MLP is applied only to the thinking tokens before injection (Zheng et al., 28 Sep 2025).

LC-FT training has two stages. In Stage 1, a frozen reference LLM processes inputs with explicit rationales, and the model aligns the pooled latent-thought representation $v^l$9 to the pooled rationale representation $\mathcal{V}_{\mathrm{abs}}$0 using cosine similarity, with stop-gradient on the reference representation. The paper notes that there is no vector quantization loss; prototypes are learned directly as parameters through alignment. In Stage 2, explicit rationales are dropped, and the model is trained with the standard language modeling objective conditioned on the thinking tokens: $\mathcal{V}_{\mathrm{abs}}$1 The trainable components are the codebook, the projection layers, the refiner, and LoRA adapters from layer $\mathcal{V}_{\mathrm{abs}}$2 onward; earlier layers are frozen. The stated aim is to train the model to use the latent thinking tokens instead of explicit CoT (Zheng et al., 28 Sep 2025).

The central computational property is that generation length at inference depends only on answer length. The reasoning prior is inserted once into hidden space rather than emitted token by token. The paper therefore describes LC-FT as a model that “thinks silently” in one pass (Zheng et al., 28 Sep 2025).

3. Adaptive compute: routing, confidence, and speed control

LC-FT is paired with GainRouter, a lightweight routing mechanism that decides whether a given input should remain in the fast codebook-guided mode or escalate to slow explicit reasoning. The router consumes a pooled question representation, the thinking tokens projected to a lower dimension, an attention-pooled summary of those thinking tokens, cosine similarity between question and hints, normalized attention entropy, and a predicted normalized length difference between fast and slow generations. The decision rule is

$\mathcal{V}_{\mathrm{abs}}$3

with labels derived from paired fast and slow runs: positive if only the slow model succeeds, negative if the fast model succeeds, and cases where both fail ignored. The stated purpose is to suppress overthinking and reduce unnecessary token generation (Zheng et al., 28 Sep 2025).

A different control mechanism appears in “Controlling Thinking Speed in Reasoning Models,” which identifies a steering vector $\mathcal{V}_{\mathrm{abs}}$4 governing the slow-fast transition and edits hidden states as

$\mathcal{V}_{\mathrm{abs}}$5

A positive $\mathcal{V}_{\mathrm{abs}}$6 pushes the model toward faster, more direct CoT; a negative $\mathcal{V}_{\mathrm{abs}}$7 toward slower, more deliberative CoT. The paper derives $\mathcal{V}_{\mathrm{abs}}$8 by PCA over hidden-state differences between fast and slow traces, and adapts $\mathcal{V}_{\mathrm{abs}}$9 token by token using a Jensen–Shannon-divergence difficulty signal between early- and late-layer next-token distributions. Across MATH-500, AIME24, AIME25, and GPQA, the adaptive scheme yields an average $\mathbf{C} \in \mathbb{R}^{M \times H},$0 Pass@1 with $\mathbf{C} \in \mathbb{R}^{M \times H},$1 tokens (Lin et al., 4 Jul 2025).

ThinkRouter places routing directly between latent and discrete reasoning spaces. Its latent backbone is Soft Thinking, where each reasoning step uses a probability-weighted average of top-$\mathbf{C} \in \mathbb{R}^{M \times H},$2 token embeddings. ThinkRouter measures confidence by $\mathbf{C} \in \mathbb{R}^{M \times H},$3 and routes to discrete token space when $\mathbf{C} \in \mathbb{R}^{M \times H},$4, and to latent space otherwise. The paper argues that low-confidence latent mixtures can introduce and propagate noise, whereas discrete selection can ground uncertain steps. ThinkRouter also uses an entropy-based Cold Stop criterion for end-of-thinking. On STEM reasoning and coding benchmarks across Qwen3-1.7B, Qwen3-8B, Qwen3-32B, and gpt-oss-20b, it reports an average improvement of $\mathbf{C} \in \mathbb{R}^{M \times H},$5 points in Pass@1 and generation-length reductions of up to $\mathbf{C} \in \mathbb{R}^{M \times H},$6 (Xu et al., 12 Feb 2026).

Taken together, these papers define a common control problem: latent reasoning is efficient, but it is not uniformly reliable. Routing, confidence estimation, and internal difficulty signals determine when compression is safe and when the model should pay the cost of explicit reasoning.

4. Architectural variants across domains and modalities

In e-commerce relevance modeling, “Thinking Broad, Acting Fast” introduces Latent Reasoning Knowledge Distillation (LRKD). The teacher uses Multi-Perspective CoT with three explicitly defined perspectives—User Intent Perspective, Structured Analysis Perspective, and Business Rule Perspective—and is trained with SFT plus DPO. The student is a BERT cross-encoder augmented with a latent reasoning extractor $\mathbf{C} \in \mathbb{R}^{M \times H},$7 that produces $\mathbf{C} \in \mathbb{R}^{M \times H},$8, aligned by MSE to a fixed sentence embedding $\mathbf{C} \in \mathbb{R}^{M \times H},$9 of the teacher rationale. The extractor is explored in MLP, Poly-Encoder, and GAT forms; the Poly-Encoder uses $c_j \in \mathbb{R}^H$0 learned context codes, which the paper explicitly interprets as a close architectural analogue to a codebook. The teacher requires $c_j \in \mathbb{R}^H$1 ms per batch of $c_j \in \mathbb{R}^H$2 pairs, whereas BERT baseline is $c_j \in \mathbb{R}^H$3 ms, LRKD-Poly is $c_j \in \mathbb{R}^H$4 ms, and LRKD-GAT is $c_j \in \mathbb{R}^H$5 ms. Offline results show, for example, AliExpress ACC/F1 improving from $c_j \in \mathbb{R}^H$6 to $c_j \in \mathbb{R}^H$7 with LRKD-GAT (Qiu et al., 29 Jan 2026).

In multimodal reasoning, CoLT replaces text CoT with a short chain of latent thoughts. A latent thought is the last-layer hidden state at a designated reasoning position, re-injected as the input embedding at the next step, and the default chain length is $c_j \in \mathbb{R}^H$8. To prevent semantically meaningless latent states and unstable training, CoLT adds a lightweight external decoder with forward supervision, latent $c_j \in \mathbb{R}^H$9 text, backward alignment, text $\mathbf{Q} \in \mathbb{R}^{K \times H}$0 latent, and an internal transition predictor over adjacent latent steps. The decoder and internal losses are removed at inference. On MMStar and MMT-Bench with Qwen3-VL-8B, text CoT uses roughly $\mathbf{Q} \in \mathbb{R}^{K \times H}$1–$\mathbf{Q} \in \mathbb{R}^{K \times H}$2 CoT tokens and $\mathbf{Q} \in \mathbb{R}^{K \times H}$3–$\mathbf{Q} \in \mathbb{R}^{K \times H}$4 seconds decoding time per sample, whereas CoLT uses $\mathbf{Q} \in \mathbb{R}^{K \times H}$5 latent tokens and $\mathbf{Q} \in \mathbb{R}^{K \times H}$6–$\mathbf{Q} \in \mathbb{R}^{K \times H}$7 seconds; the reported speedups are $\mathbf{Q} \in \mathbb{R}^{K \times H}$8 overall and $\mathbf{Q} \in \mathbb{R}^{K \times H}$9 in text decoding time, with accuracy improving from $K$0 to $K$1 on MMStar and from $K$2 to $K$3 on MMT-Bench (Hu et al., 30 Jun 2026).

Abstract-CoT is the discrete counterpart. It adds a reserved abstract vocabulary $K$4, typically with codebook size $K$5, and inserts a latent reasoning segment

$K$6

with $K$7. Training uses a policy-iteration-style warm-up loop alternating between bottlenecked SFT from verbal CoT and self-distillation from prompts alone, followed by warm-started GRPO under constrained decoding. Reported results include $K$8 tokens on MATH-500 for Qwen3-8B, $K$9 on AlpacaEval-LC-2.0, and $16$0 on HotpotQA, corresponding to up to $16$1 fewer reasoning tokens while maintaining comparable performance (Ramji et al., 24 Apr 2026).

These variants differ substantially in implementation, but all preserve the same core pattern: broad reasoning supervision at training time, compact latent computation at inference time.

5. Empirical profile, interpretability, and precursors

LC-FT reports quantitative gains on both mathematical reasoning and programming. On AIME, Qwen3-Thinking reaches $16$2 with $16$3 tokens, LC-FT alone reaches $16$4 with $16$5 tokens, and LC-FT + GainRouter reaches $16$6 with $16$7 tokens. On OlympiadBench, the corresponding values are $16$8 with $16$9 tokens for Qwen3-Thinking, $48$0 with $48$1 tokens for LC-FT, and $48$2 with $48$3 tokens for LC-FT + GainRouter. On HumanEval, Qwen3-Thinking is $48$4 at $48$5 tokens, LC-FT is $48$6 at $48$7 tokens, and LC-FT + GainRouter is $48$8 at $48$9 tokens. The paper also reports sparse prototype usage, low normalized entropy, and class-consistent prototype reuse, and gives a qualitative case in which the model emphasizes a prototype cluster coherent with “logarithmic simplification + telescoping” (Zheng et al., 28 Sep 2025).

Longer-term antecedents show that latent codebooks were already being used to compress search spaces and to make internal states more discrete. Latent Programmer learns a VQ-style discrete latent sequence $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$0 for programs and performs two-level search: first over latent plans, then over programs conditioned on those plans. On the string transformation DSL, it reports $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$1 accuracy at beam sizes $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$2, compared with $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$3 for RobustFill [Transformer]. On Python code generation, it reports BLEU $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$4 at beam sizes $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$5, compared with $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$6 for RobustFill [Transformer]. The latent sequence is presented as a compact plan that makes search more efficient (Hong et al., 2020).

“Codebook Features” pushes the codebook idea inside the network at every layer. Hidden activations are replaced by sums of the top-$$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$7 codes from a learned codebook, yielding sparse, discrete internal features. The paper emphasizes that $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$8 is typically less than $$\mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H},$$9 of the codebook size, computes channel capacity as $v^l$00, and reports, for example, $v^l$01 average precision for codebook features versus $v^l$02 for the best neuron at matched recall in the FSM setting. It also shows that activating topic-linked codes can steer generation toward topics such as “dragon,” “fire,” or “football” (Tamkin et al., 2023).

These precursors clarify that latent codebooks serve two roles. They are an efficiency mechanism, because they reduce search or generation complexity, and they are a representational mechanism, because they make reasoning states more modular, sparse, and sometimes more controllable.

6. Limitations, misconceptions, and future directions

A common misconception is that latent fast-thinking methods are uniformly discrete. The current literature is mixed. LC-FT uses a discrete codebook in index space but forms continuous thinking vectors through soft attention; ThinkRouter’s latent steps are soft mixtures of token embeddings; LRKD and CoLT operate in continuous latent spaces; Abstract-CoT uses explicitly discrete abstract tokens (Zheng et al., 28 Sep 2025, Xu et al., 12 Feb 2026, Hu et al., 30 Jun 2026, Ramji et al., 24 Apr 2026).

A second misconception is that latent reasoning simply dominates explicit CoT. The papers do not support that conclusion. LC-FT notes that some tasks still need explicit CoT, especially the hardest reasoning tasks or tasks requiring very long derivations; GainRouter exists precisely because fast mode is not always sufficient. LRKD notes an interpretability trade-off: the teacher’s CoT is fully interpretable, whereas the student’s latent reasoning is explainable only indirectly. CoLT still depends on explicit CoT annotations during training, and its fixed $v^l$03 can underfit hard problems or introduce redundant constraints when too large. Abstract-CoT reports that cold-start RL from random abstract embeddings is ineffective and that a short abstract budget can remain a bottleneck on the hardest math problems (Zheng et al., 28 Sep 2025, Qiu et al., 29 Jan 2026, Hu et al., 30 Jun 2026, Ramji et al., 24 Apr 2026).

ThinkRouter adds a further caution specific to soft latent mixtures: aggregated low-confidence alternatives can introduce and propagate noise, producing high confidence in unreliable trajectories. That observation motivates confidence-aware routing and suggests that latent compression is safest when the model’s token distribution is already sufficiently sharp (Xu et al., 12 Feb 2026).

The future directions described across these papers are convergent. LC-FT explicitly suggests latent codebook designs closer to VQ-VAE style vector quantization, explicit clustering of strategies, better routing policies, and controllable reasoning by targeting specific codebook entries or regions. LRKD points toward discrete latents or mixture-of-experts gating based on perspectives. CoLT points toward adaptive numbers of latent steps and more structured latent spaces or codebooks. Abstract-CoT points toward budget-adaptive abstract reasoning and hierarchical codebooks (Zheng et al., 28 Sep 2025, Qiu et al., 29 Jan 2026, Hu et al., 30 Jun 2026, Ramji et al., 24 Apr 2026).

The resulting research program is clear: retain the strategic benefits of CoT, but move the reasoning substrate from long natural-language traces to compact latent variables that can be injected, decoded, routed, or discretely selected. In that sense, latent codebooks for fast thinking define not one architecture but a family of methods for shifting reasoning from verbose external text into shorter internal representations, with efficiency gains that are now documented across language reasoning, coding, retrieval-style relevance modeling, and multimodal reasoning.