Activation-space CoT Vectors

Updated 16 April 2026

Activation-space CoT vectors are continuous latent representations that encode intermediate reasoning states in high-dimensional spaces for efficient and flexible processing.
They employ gradient-based inference and iterative rethinking to refine reasoning steps, leading to notable performance gains over explicit CoT methods.
Empirical studies show that with fewer parameters, these methods achieve comparable or superior accuracy on benchmarks like GSM8K and arithmetic tasks.

A continuous activation-space Chain-of-Thought (CoT) vector is a high-dimensional latent representation encoding an intermediate reasoning state, abstracted away from explicit language tokens and embedded directly in a model’s hidden activation space. This paradigm aims to factorize and compress reasoning steps into continuous vectors—distinct from the discrete tokens used in explicit CoT—enabling multi-step, interpretable, and computation-efficient reasoning in LLMs as well as multimodal and action-oriented architectures. Activation-space CoT vectors support iterative refinement, gradient-based inference over reasoning strategies, and flexible control over exploration and execution trade-offs.

1. Formal Foundations and Model Factorization

Activation-space CoT vectors are defined as continuous latent vectors, typically denoted $z \in \mathbb{R}^d$ , which parameterize “what to reason about” independently from “how to verbalize it.” The generative framework typically factorizes the joint model over an observable reasoning trace $x = (x_1, ..., x_n)$ and its associated latent $z$ as: $p(x, z) = p(z) \, p(x \mid z)$ where $p(z)$ is a learned prior over the latent thought vector, and $p(x \mid z)$ represents an autoregressive decoder that emits the reasoning trace conditioned on $z$ (Kong et al., 6 Feb 2026). Marginalizing out the latent yields $p(x) = \int p(z) p(x|z) dz$ , establishing a continuous latent bottleneck for the entire reasoning process.

Decoder architectures adopt standard Transformer decoder blocks augmented with cross-attention over $z$ at every layer, such that each token $x_t$ is generated as: $x = (x_1, ..., x_n)$ 0 This enforces that long-range or global dependencies in the output must be coordinated via the latent vector, ensuring that $x = (x_1, ..., x_n)$ 1 carries the high-level plan (Kong et al., 6 Feb 2026).

2. Inference-Time Optimization and Gibbs-Style Rethinking

Inference in the activation-space CoT vector framework decouples the generative trace from the latent structure, allowing for test-time optimization of the reasoning plan:

Given input $x = (x_1, ..., x_n)$ 2, a prior $x = (x_1, ..., x_n)$ 3 maps isotropic Gaussian noise $x = (x_1, ..., x_n)$ 4 into the latent manifold via a learned transport encoder $x = (x_1, ..., x_n)$ 5.
During “rethinking” inference, the procedure alternates between:
1. Generation: Decode a candidate reasoning trace $x = (x_1, ..., x_n)$ 6,
2. Reflection/Optimization: Update $x = (x_1, ..., x_n)$ 7 (and thus $x = (x_1, ..., x_n)$ 8) by gradient ascent:
$x = (x_1, ..., x_n)$ 9

Equivalent updates in $z$ 0-space take the form:

$z$ 1

This Gibbs-style process is repeated for $z$ 2 rounds, with the best trace (as measured by joint log-probability) retained (Kong et al., 6 Feb 2026).

This optimization over the continuous manifold allows for recovery from early reasoning errors, iterative self-correction, and more effective adaptation to each input.

3. Prior Learning, Training Objectives, and Variational Structure

The latent prior is parameterized either as a transport map (e.g., Transformer encoder over noise) or as a variational family shaped during training. Both prior and decoder are trained by maximizing the stochastic variational evidence lower bound (ELBO): $z$ 3 Here, $z$ 4 can be a variational Gaussian, optimized per instance (non-amortized), and global model parameters are updated at a slower rate. This objective aligns reasoning traces with robust regions of the latent manifold, regularizing $z$ 5 away from degenerate or collapsed solutions (Kong et al., 6 Feb 2026).

4. Properties and Empirical Performance

Empirical results on GSM8K and arithmetic benchmarks demonstrate that activation-space CoT vector models significantly outperform explicit-CoT baselines with tenfold fewer parameters:

A $z$ 6B param model, with 30 inference-time rethinking iterations, matches or exceeds a 3B parameter explicit-CoT baseline $z$ 7 vs $z$ 8 on GSM8K.
Out-of-domain tasks such as SVAMP and MultiArith similarly show $z$ 9– $p(x, z) = p(z) \, p(x \mid z)$ 0 point absolute gains over larger explicit baselines.
Accuracy increases monotonically with the number of inference rethinking steps, confirming that gradient-based search in activation-space produces strictly improving solutions under the ELBO.

The separation between “plan” ( $p(x, z) = p(z) \, p(x \mid z)$ 1) and “verbalization” (decoder) yields a highly parameter-efficient model and enables effective navigation of the manifold of reasoning strategies.

5. Comparison to Other Continuous Latent CoT Paradigms

Activation-space CoT vectors are a highly general instantiation of latent reasoning. Compared to direct recurrence approaches (e.g., COCONUT (Hao et al., 2024), CODI (Liang et al., 31 Jan 2026), PLUME (He et al., 2 Apr 2026)), the present framework’s learned latent prior and inference-time optimization offer:

Stronger abstraction: $p(x, z) = p(z) \, p(x \mid z)$ 2 can represent entire reasoning programs or “plans,” rather than token-level continuations,
Improved exploration: Optimization over $p(x, z) = p(z) \, p(x \mid z)$ 3 at test time enables the model to re-express or refine its internal strategy after seeing the result of a decoding attempt,
Algorithmic flexibility: The framework accommodates various exploration–execution trade-offs, as analyzed through symbolic indices and margin-based error guarantees (Zou et al., 1 Feb 2026).

However, the approach requires careful per-instance optimization and regularization to avoid latent collapse; empirical effectiveness depends on alignment of the learned manifold with the distribution of valid reasoning strategies.

6. Mechanistic Interpretation and Limitations

Activation-space CoT vectors encode a compressed, declarative representation of the reasoning plan. Long-range dependencies and errors in early steps can be corrected through explicit optimization in the latent space, a property absent from single-shot explicit CoT.

Nevertheless, step-by-step mechanistic studies (Liang et al., 31 Jan 2026) show that, for deeply sequential or “non-compressible” tasks, activation-space CoT can degenerate to shortcuts (tracking only final-step intermediates), or fuse explicit and latent trajectories only at the answer position. This failure mode is particularly pronounced when the latent budget is insufficient for full explicit rollout or when underlying algebraic structure resists contraction.

Statistical and theoretical results demonstrate that the effectiveness of latent reasoning depends both on the compressibility of the reasoning chain and on the shape of the latent prior; adaptive budgets and per-step supervision improve robustness but add training complexity (Liang et al., 31 Jan 2026, Chen et al., 22 May 2025).

7. Broader Implications and Future Directions

The introduction of activation-space CoT vectors motivates several emerging research directions:

Generalization and abstraction: The latent manifold can support transfer of reasoning patterns across tasks, input domains, or even modalities if appropriately regularized.
Inference efficiency: Continuous vectors compress reasoning trajectories—hundreds of tokens into single or tens of vector updates—yielding up to $p(x, z) = p(z) \, p(x \mid z)$ 4 inference speedup in multimodal settings (He et al., 2 Apr 2026).
Adaptive reasoning: Curriculum learning and variational inference over $p(x, z) = p(z) \, p(x \mid z)$ 5 support dynamic adaptation according to task hardness, exploration requirements, or prior uncertainty metrics.
System design: By decoupling abstract reasoning from surface-level narration, these methods enable transfer, modularity, and interpretability in LLM-based agents.

Open challenges remain in scaling latent budgets adaptively, ensuring full-step supervision in deeply sequential domains, and probing or visualizing the geometry and semantics of the learned latent manifold.

In summary, activation-space CoT vectors provide a theoretically principled and empirically validated method for compact, abstract, and iterative reasoning within LLMs and related architectures. By factorizing the reasoning process into an optimizable continuous latent bottleneck and leveraging advanced inference-time procedures, they enable models to reason more efficiently and adaptively than traditional token-based chains of thought (Kong et al., 6 Feb 2026).