Latent Thought Embeddings

• Latent thought embeddings are compact intermediate representations that bridge perception, context integration, and output generation.
• They encompass diverse families including sequence-level predictive, inline latent tokenization, recurrent reasoning, bottleneck methods, and semantic state conditioning.
• They improve model performance across multimodal tasks while posing challenges in interpretability and computational trade-offs.

Latent thought embeddings are intermediate internal representations that mediate between perception, context integration, and output generation. In the recent literature, the term denotes several related objects: predictive embeddings of expert multimodal trajectories, inline latent “sketches” inside chain-of-thought, recurrent chunk-level reasoning states, fixed-capacity bottleneck tokens for retrieval, topic-conditioned embeddings tied to latent semantic states, and explicit sequence-level latent variables endowed with priors (Adhikari et al., 9 Apr 2026, Shao et al., 31 Jan 2026, Amos et al., 9 Feb 2026, Wu et al., 11 Jun 2026, Kong et al., 3 Feb 2025). Across these formulations, the common claim is not that reasoning must be verbalized token by token, but that useful intermediate computation can be carried by compact latent states that are continuous, discrete, or hybrid, and then consumed by downstream decoders, retrievers, or control policies.

1. Definitions and conceptual scope

The literature does not present a single canonical object called a latent thought embedding. Instead, it describes a family of intermediate semantic states. In Pearl, latent thought embeddings are “compact, continuous representations of expert tool-use trajectories” that summarize the “thinking” a model would have done with external tools; formally, for input $x=\langle I_0,Q\rangle$ and expert trajectory $R=(I_1,T_1,\dots,I_N,T_N)$, Pearl computes $h_x=\mathrm{Enc}(x)$, $h_R=\mathrm{Enc}(R)$, and a predictor output $\hat{h}_R=\mathrm{Pred}(h_x)$ (Adhikari et al., 9 Apr 2026). In Thinking States, the relevant object is a chunk-aligned latent state $S_i\in\mathbb{R}^{c\times d}$, obtained by compressing generated thought tokens and additively injecting them into the next chunk’s shallow-layer embeddings (Amos et al., 9 Feb 2026). In LaME, latent thought is instantiated as $K$ learnable reason tokens appended to the multimodal input, forming a fixed-capacity bottleneck for single-pass embedding-oriented reasoning (Wu et al., 11 Jun 2026).

Other works define the object more directly through modality. “Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings” treats latent thought embeddings as compact visual carriers or “sketches” represented in the VLM’s native embedding space and interleaved with text (Shao et al., 31 Jan 2026). MCOUT defines a thought as a continuous hidden vector of shape $B\times1\times D$, iteratively refined and appended to the multimodal sequence (Pham et al., 18 Aug 2025). FLAIR defines “silent thought” during listening as $z_t=\mathrm{Softmax}(y_{\text{txt},t})E$, a weighted average of vocabulary embeddings that recursively feeds the next step (Wu et al., 18 Mar 2026).

The conceptual scope also includes older antecedents in which latent thought is not framed as chain-of-thought compression but as latent semantic state conditioning. STE learns topic-specific word embeddings $U_{w,z}$ and $R=(I_1,T_1,\dots,I_N,T_N)$0, so that a word occurrence can be represented by a posterior-weighted mixture $R=(I_1,T_1,\dots,I_N,T_N)$1 or by a MAP topic-specific embedding (Shi et al., 2017). “Learning with Memory Embeddings” formalizes shared latent vectors for entities, predicates, and time across semantic, episodic, and sensory memory, and explicitly treats time-latent embeddings as summaries of sensory context that can be decoded into symbolic content (Tresp et al., 2015). Taken together, these works define latent thought embeddings less by a fixed architecture than by a role: an intermediate state that carries compressed, task-relevant structure across a computation.

2. Representational families

One major family is sequence-level predictive embedding. Pearl learns a holistic embedding of an entire expert multimodal trajectory rather than reconstructing intermediate images token by token. The target $R=(I_1,T_1,\dots,I_N,T_N)$2 encodes both visual edits and intervening reasoning text, and the predictor aligns $R=(I_1,T_1,\dots,I_N,T_N)$3 to a stop-gradient target with a JEPA-style loss. This makes the latent thought a sequence summary of multi-step tool use rather than a decoded latent canvas (Adhikari et al., 9 Apr 2026).

A second family is inline latent tokenization. In modal-mixed CoT, text tokens are interleaved with latent visual blocks bracketed by $R=(I_1,T_1,\dots,I_N,T_N)$4 and $R=(I_1,T_1,\dots,I_N,T_N)$5; each latent token is a $R=(I_1,T_1,\dots,I_N,T_N)$6-dimensional embedding, and a block is produced by a diffusion-based latent head conditioned on the LLM’s hidden states (Shao et al., 31 Jan 2026). RoT renders textual CoT into single-line images and uses a frozen vision encoder to produce target visual embeddings $R=(I_1,T_1,\dots,I_N,T_N)$7; the LLM then learns projected latent visual tokens $R=(I_1,T_1,\dots,I_N,T_N)$8 aligned to those targets (Wang et al., 21 Jan 2026). LT-Tuning constructs a latent token $R=(I_1,T_1,\dots,I_N,T_N)$9 by Context-Prediction Fusion,

$h_x=\mathrm{Enc}(x)$0

or, in untied models, by fusing $h_x=\mathrm{Enc}(x)$1 with an adapted hidden state, thereby treating the latent thought as a next-input embedding on the model’s own embedding manifold (Liu et al., 10 Feb 2026). FLAIR’s latent update is similar in spirit but softer: the model turns vocabulary probabilities into a continuous latent input during listening rather than generating explicit reasoning text (Wu et al., 18 Mar 2026).

A third family is recurrent state injection. Thinking States decodes natural-language thought tokens $h_x=\mathrm{Enc}(x)$2 from a deep layer, compresses them through a causal Transformer encoder into $h_x=\mathrm{Enc}(x)$3, and injects the result additively into the next chunk, $h_x=\mathrm{Enc}(x)$4 (Amos et al., 9 Feb 2026). The gloss-free sign language system “Think in Latent Thoughts” maintains $h_x=\mathrm{Enc}(x)$5 learnable thought slots $h_x=\mathrm{Enc}(x)$6, refines them across $h_x=\mathrm{Enc}(x)$7 thinking layers with causal self-attention and routed cross-attention over frame evidence, and uses them as an ordered latent chain between video and text (Jiang et al., 16 Apr 2026). MCOUT also fits the recurrent pattern: the LM’s last hidden state, or its visually attended variant, is iteratively appended as the current thought vector and then recomputed (Pham et al., 18 Aug 2025).

A fourth family is bottleneck reasoning. LaME appends $h_x=\mathrm{Enc}(x)$8 learnable reason tokens and an [EMBED] token to the multimodal input; the first $h_x=\mathrm{Enc}(x)$9 reason tokens feed a decoder probe, the remaining $h_R=\mathrm{Enc}(R)$0 are mean-pooled into a reason embedding, and the [EMBED] token yields the final retrieval vector (Wu et al., 11 Jun 2026). LTMs similarly use explicit latent vectors $h_R=\mathrm{Enc}(R)$1, where each $h_R=\mathrm{Enc}(R)$2 is a set of thought vectors injected into decoder layer $h_R=\mathrm{Enc}(R)$3 via cross-attention; here the latent thought is not a recurrent state in token space but a sequence-level global variable inferred per example (Kong et al., 3 Feb 2025).

A fifth family is latent semantic state conditioning. STE’s topic variable $h_R=\mathrm{Enc}(R)$4 is discrete rather than continuous, but it plays the same structural role: it conditions the embedding of a word occurrence, resolves polysemy, and can be interpreted as a latent “thought state” over semantic context (Shi et al., 2017). This suggests that the latent-thought perspective is compatible with both continuous bottlenecks and posterior-weighted discrete state mixtures.

3. Objectives and inference regimes

The training objectives vary sharply across systems, and these differences determine whether latent thought embeddings are predictive, reconstructive, recurrent, or variational. Pearl uses

$h_R=\mathrm{Enc}(R)$5

with $h_R=\mathrm{Enc}(R)$6, SmoothL1 distance, no contrastive negatives, and a next-latent regularizer that encourages hidden states to behave as belief states (Adhikari et al., 9 Apr 2026). The latent target is never reconstructed at inference; the model answers directly with standard VLM decoding.

Modal-mixed CoT instead trains a joint next-token and diffusion $h_R=\mathrm{Enc}(R)$7-prediction objective, reconstructing the VLM’s own intermediate vision embeddings, and then applies GRPO reinforcement learning to learn when to switch modalities and how to compose longer mixed chains (Shao et al., 31 Jan 2026). RoT adopts a two-stage regime: Stage I aligns projected LLM hidden states to frozen visual embeddings with MSE, $h_R=\mathrm{Enc}(R)$8, together with answer prediction; Stage II freezes the projection head and fine-tunes the LLM with LoRA to autoregressively navigate the visual latent manifold (Wang et al., 21 Jan 2026). LT-Tuning uses next-token cross-entropy throughout all three stages, relying on latent construction rather than auxiliary regularization to stabilize reasoning (Liu et al., 10 Feb 2026).

Thinking States is supervised with natural-language thought sequences aligned to chunks. Teacher-forced state injection permits all chunks to be processed in parallel, and the joint loss is

$h_R=\mathrm{Enc}(R)$9

so the model learns interpretable thought tokens while avoiding BPTT (Amos et al., 9 Feb 2026). FLAIR adopts a conditional ELBO in which the reconstruction term supervises speaking, a KL term aligns the listening-phase soft vocabulary distribution to a non-causal Expert posterior, and a timing loss controls think/speak switching; the total objective is $\hat{h}_R=\mathrm{Pred}(h_x)$0 with $\hat{h}_R=\mathrm{Pred}(h_x)$1 and $\hat{h}_R=\mathrm{Pred}(h_x)$2 (Wu et al., 18 Mar 2026). LTMs likewise optimize an ELBO,

$\hat{h}_R=\mathrm{Pred}(h_x)$3

using dual-rate optimization: fast local updates for sequence-specific variational parameters and slow global updates for decoder parameters (Kong et al., 3 Feb 2025).

LaME formulates latent reasoning as a weakly supervised information bottleneck. The reason tokens are shaped by a decoder probe, a contrastive embedding head, and a diversity regularizer, then tied to the production embedding through InfoNCE on the [EMBED] token (Wu et al., 11 Jun 2026). At inference, however, LaME remains single-pass: no textual CoT decoding, no iterative latent rollout, and no auxiliary decoder invocation are required. This stands in contrast to the inference-time rethinking model for math, where a learned prior maps Gaussian noise into latent thought vectors and a Gibbs-style loop alternates between generating a trace and optimizing the latent to better explain it; with this design, latent thought becomes an object of explicit test-time optimization rather than only a training-time auxiliary state (Kong et al., 6 Feb 2026).

4. Domains and reported performance

In multimodal perception and reasoning, Pearl reports that on Qwen2.5-VL-7B-Instruct and LVR data, Pearl versus LVR (4 steps) achieves V* 81.5 vs 80.1 and MMVP 73.5 vs 72.0, while Pearl versus SFT yields V* 81.5 vs 79.1 and MMVP 73.5 vs 65.7; on ThinkMorph data, Pearl versus SFT gives V* 73.8 vs 42.4 and MMVP 75.3 vs 36.7 (Adhikari et al., 9 Apr 2026). Modal-mixed CoT reports “Ours (SFT) average 26.7 (best)” on vision-intensive reasoning and, in efficiency analysis, “our latent steps take 3.10s with 50 denoising steps,” compared with “tool-use (Thyme) single call is 8.36s” and “text-only is 1.03s” (Shao et al., 31 Jan 2026). MCOUT improves a 1B baseline on ScienceQA from 56.17% to 58.86% with MCOUT-Base at $\hat{h}_R=\mathrm{Pred}(h_x)$4, and on MMMU from 25.44% to 27.53% with MCOUT-Base at $\hat{h}_R=\mathrm{Pred}(h_x)$5 (Pham et al., 18 Aug 2025). RoT compresses explicit CoT from 108.4 tokens to 32.0 on grade-school tasks and from 291.5 to 64.0 on MATH, while reducing GSM-Hard inference time from 8.55s to 1.84s on a single NVIDIA H20 GPU (Wang et al., 21 Jan 2026).

In retrieval, LaME reports MMEB-V2 All = 64.4 at ~2B and All = 68.8 at ~7B, with MRMR All = 49.8 at ~7B; it further reports “~60× faster inference than explicit CoT approaches,” “~2× faster than iterative latent rollout baselines,” and throughput of “~6.2 ± 0.3 samples/s with K=8 on a single GPU” (Wu et al., 11 Jun 2026). In language modeling, LTMs report validation perplexity 5.58 for LTM-Large on OpenWebText and strong zero-shot perplexity upper bounds across PTB, WikiText, LM1B, LAMBADA, AG News, PubMed, and Arxiv, while conditional generation reaches MAUVE = 0.974 for LTM-Large with multinomial sampling (Kong et al., 3 Feb 2025).

In language reasoning, Thinking States matches CoT on 2-Hop QA while improving latency: “Full Knowledge: Thinking States 54.91 (×1.19) vs CoT 54.79 (×1)” and “Parametric Knowledge: Thinking States 43.05 (×1.23) vs CoT 43.07 (×1)” (Amos et al., 9 Feb 2026). On state-tracking, it reports Parity at $\hat{h}_R=\mathrm{Pred}(h_x)$6: Thinking States 100.00 vs CoT 64.38 vs No-CoT 59.60, and Variable Assignment at $\hat{h}_R=\mathrm{Pred}(h_x)$7: Thinking States 97.71 vs CoT 87.75 vs No-CoT 2.19 (Amos et al., 9 Feb 2026). LT-Tuning reports average scores of 36.4 at 1B, 52.4 at 3B, and 70.3 at 8B with adapter, outperforming latent baselines and mitigating the severe degradation shown by Coconut at 8B (Liu et al., 10 Feb 2026).

In speech and sign language, FLAIR improves LlamaQ from 73.0 to 78.0 and MMSU from 50.2 to 56.2 while maintaining competitive full-duplex interaction metrics; on the Impatient dataset it reports turn-taking latency 0.39 s and barge-in latency 0.46 s with latent thinking (Wu et al., 18 Mar 2026). “Think in Latent Thoughts” reports BLEU-4 27.22 on PHOENIX14T, 23.92 on CSL-Daily, 13.39 on How2Sign, 19.55 on OpenASL, and 30.22 on LC-HKSLT with in-domain pretraining and fine-tuning (Jiang et al., 16 Apr 2026). In multiagent collaboration, ThoughtComm reports, for Qwen 3-1.7B on MATH, 93.00% ± 1.14 versus MAFT 75.80% ± 1.92 versus single 43.60% ± 2.22, and on GSM8K 85.00% ± 1.60 versus 84.20% ± 1.63 versus 67.40% ± 2.10 (Zheng et al., 23 Oct 2025).

5. Interpretability, superposition, and debate

A central controversy concerns what latent thought embeddings actually represent. Pearl argues that reconstruction-based latent reasoning methods may not be simulating image edits in latent space. In the reported analysis, “>75% of supervised edited images yield >8 latent tokens during training, yet inference fixes 4–8 tokens,” while accuracy is largely insensitive to token count and correlations are near-zero or negative, including BLINK $\hat{h}_R=\mathrm{Pred}(h_x)$8, MMVP $\hat{h}_R=\mathrm{Pred}(h_x)$9, and average $S_i\in\mathbb{R}^{c\times d}$0 (Adhikari et al., 9 Apr 2026). The paper interprets this as evidence that such systems “primarily learn helpful embeddings rather than performing genuine image-edit simulation in latent space.”

A related dispute concerns superposition. “The Illusion of Superposition?” studies training-free convex combinations, fine-tuned latent thoughts, and from-scratch latent-thought models, and reports that only from-scratch models with constrained capacity show signatures of superposition; in training-free and fine-tuned regimes, superposition collapses or is not used, and shortcut solutions dominate (Rizvi-Martel et al., 7 Apr 2026). The depth sweep is especially pointed: with latents, accuracy falls from 94.5% at 2 layers to 61.6% at 12 layers, while without latents it rises from 13.8% to 63.0%, indicating that higher-capacity models increasingly bypass latent computation (Rizvi-Martel et al., 7 Apr 2026). This suggests that “latent thought” should not automatically be read as “multiple hypotheses maintained in superposition.”

Interpretability remains uneven across the literature. RoT explicitly addresses analyzability by supervising latent states with rendered CoT images and visualizing vision-embedding heatmaps and token-similarity matrices (Wang et al., 21 Jan 2026). By contrast, Pearl states that its embeddings are “continuous and not directly interpretable,” and LaME notes that it has “No explicit CoT; latent reasoning is less interpretable than textual traces” (Adhikari et al., 9 Apr 2026, Wu et al., 11 Jun 2026). The current evidence therefore supports a narrow conclusion: latent thought embeddings can improve reasoning efficiency and performance, but they are not uniformly transparent, nor do they uniformly correspond to explicit intermediate world models.

6. Limitations and future directions

The limitations recur across otherwise different systems. Pearl depends on “high-quality expert tool-use sequences,” incurs two independent forward passes per example during training, and does not explicitly plan actions in latent space at test time (Adhikari et al., 9 Apr 2026). Modal-mixed CoT identifies latent hallucination, sensitivity to START/END overuse, computational overhead from diffusion, and latent token budget sensitivity (Shao et al., 31 Jan 2026). Thinking States requires chunk-aligned thought supervision and shows an accuracy gap on GSM8K-style tasks, where 42.22 still lags CoT at 60.50; it is also sensitive to query structure, with prepending the final question improving difficult cases from 42.22 to 48.65 (Amos et al., 9 Feb 2026). LT-Tuning reports large drops when fusion is removed, especially at 8B, indicating that poorly constructed latent tokens can be worse than none (Liu et al., 10 Feb 2026).

Capacity and supervision quality are also recurring constraints. LaME warns that too many reason tokens can weaken the bottleneck and slightly harm standard retrieval, and that long CoT decoding targets dilute information-bottleneck gradients (Wu et al., 11 Jun 2026). LTMs note that excessive fast inference steps can make the variational posterior overconfident and degrade decoder learning (Kong et al., 3 Feb 2025). The inference-time rethinking framework for math explicitly notes that its likelihood-based reflection relies on near-expert demonstrations and may converge to high-likelihood but incorrect traces under noisy supervision (Kong et al., 6 Feb 2026).

The forward-looking proposals are correspondingly diverse. Pearl suggests using learned embeddings as latent world models for explicit multi-step planning without tools (Adhikari et al., 9 Apr 2026). Modal-mixed CoT proposes extension to audio, 3D, and action latents (Shao et al., 31 Jan 2026). RoT raises adaptive rendering, contrastive alignment, temporal consistency, and adaptive budgets as open directions (Wang et al., 21 Jan 2026). LaME points toward low-latency retrieval systems that retain reasoning-driven structure without CoT annotations (Wu et al., 11 Jun 2026). ThoughtComm extends the concept beyond single-model reasoning to multiagent settings, where shared and private latent thoughts can be structurally recovered and routed between agents (Zheng et al., 23 Oct 2025). A plausible implication is that latent thought embeddings are becoming less a single method than a general design space: intermediate representations can be optimized for prediction, retrieval, control, collaboration, or grounding, provided that the latent interface is matched to the supervision, bottleneck, and inference regime of the task.