--- title: Latent Codebooks for Fast Thinking url: https://www.emergentmind.com/topics/latent-codebooks-for-fast-thinking type: topic --- # Latent Codebooks for Fast Thinking Searching arXiv for the cited papers to ground the article in current literature. Latent codebooks for fast thinking are mechanisms that replace, compress, or control explicit Chain-of-Thought (CoT) with internal latent representations so that a model can preserve strategy-level guidance while reducing autoregressive reasoning length, latency, and token cost. In the most direct formulation, a model learns a finite bank of reasoning prototypes from concise CoT sketches and, at inference, conditions on a small set of latent “thinking vectors” rather than generating a long rationale; related work broadens the idea to steering directions, latent reasoning extractors, short chains of latent thoughts, and reserved abstract reasoning vocabularies [2509.23633] [2507.03704] [2604.22709]. ## 1. Conceptual scope and terminology The immediate motivation is the inefficiency of explicit reasoning traces. Reasoning-oriented LLMs often rely on CoT, self-consistency, or tree-of-thought style decoding, but long reasoning traces increase wall-clock time and token usage, and can also induce “overthinking,” where extra steps introduce irrelevant detours and surface-level heuristics that degrade accuracy. In the LC-FT study, Qwen3-Thinking uses approximately \(19\)k tokens on AIME, which is presented as a concrete example of this cost regime [2509.23633]. Within this literature, “fast thinking” denotes inference in which the model produces the answer directly, guided by a compact internal representation of reasoning strategy rather than by explicit textual steps. “Slow thinking” denotes explicit multi-step generation in token space. The distinction appears both in models that learn reusable strategy priors and in models that dynamically modulate reasoning speed at test time [2509.23633] [2507.03704]. The term “codebook” is used in more than one sense. In LC-FT, the codebook is a learnable matrix of discrete strategy prototypes, indexed discretely but accessed through soft attention, yielding continuous mixtures [2509.23633]. In “Controlling Thinking Speed in Reasoning Models,” a single layerwise steering vector \(v^l\) functions as a one-dimensional latent code for the slow-fast transition, and the paper explicitly motivates extending that single axis into a set of reasoning-mode directions [2507.03704]. In “Abstract Chain-of-Thought,” the codebook is a reserved abstract vocabulary \(\mathcal{V}_{\mathrm{abs}}\) of discrete non-linguistic tokens generated under constrained decoding [2604.22709]. In “CoLT,” by contrast, there is no discrete VQ-style codebook; the latent units are continuous hidden states placed at designated reasoning positions [2606.31986]. This suggests a useful working distinction. A **narrow** usage refers to finite prototype banks or reserved vocabularies. A **broader** usage refers to any compact latent intermediate representation that mediates reasoning and can be controlled, selected, or routed more efficiently than full verbal CoT. ## 2. Latent Codebooks for Fast Thinking in large language models LC-FT defines the codebook as \[ \mathbf{C} \in \mathbb{R}^{M \times H}, \] where each row \(c_j \in \mathbb{R}^H\) is a strategy prototype. A small, fixed set of learnable queries \[ \mathbf{Q} \in \mathbb{R}^{K \times H} \] retrieves mixtures of these prototypes by attention, with \(K\) much smaller than the number of tokens in a CoT trace, and the paper gives \(16\)–\(48\) as representative values. The resulting latent hints are the thinking tokens \[ \mathbf{T}=\mathbf{A}(\mathbf{C}\mathbf{W}_v)\mathbf{W}_o \in \mathbb{R}^{K\times H}, \] where \(\mathbf{A}\) is the attention matrix over codebook entries [2509.23633]. The training signal does not use full verbose rationales. Instead, the method constructs concise CoT sketches \(\tilde r\). A teacher model proposes a candidate hint, and the hint is accepted only if it does not leak the answer and if the backbone model, when prompted with the question plus hint, produces an answer that a task-specific verifier accepts. The resulting dataset is \[ \mathcal{D} = \{ (x_i, \tilde r_i, y_i) \}_{i=1}^N. \] These sketches are used only during training, not at inference [2509.23633]. The thinking tokens are injected at an internal transformer layer \(L\). If the text input has hidden states \(\{z_1,\dots,z_n\}\) at layer \(L\), LC-FT appends the latent slots: \[ \mathbf{Z}^{(L)}=\big[z_1,\dots,z_n;\;\mathbf{t}_1,\dots,\mathbf{t}_K\big] \in \mathbb{R}^{(n+K)\times H}. \] The injected vectors are not input tokens; they are non-textual latent slots. They are masked before layer \(L\), so earlier layers process only the original token sequence, and from layer \(L\) onward the model attends jointly over text states and thinking tokens. A small refiner MLP is applied only to the thinking tokens before injection [2509.23633]. LC-FT training has two stages. In Stage 1, a frozen reference LLM processes inputs with explicit rationales, and the model aligns the pooled latent-thought representation \(z_T^{(l)}\) to the pooled rationale representation \(z_R^{(l)}\) using cosine similarity, with stop-gradient on the reference representation. The paper notes that there is no vector quantization loss; prototypes are learned directly as parameters through alignment. In Stage 2, explicit rationales are dropped, and the model is trained with the standard language modeling objective conditioned on the thinking tokens: \[ \mathcal{L}_{\mathrm{LM}} = -\sum_{t=1}^{|y|} \log p_\theta\!\left(y_t\,\middle|\,y_{