Multimodal Latent Reasoning

• Multimodal latent reasoning is a method that performs inference using continuous hidden representations, preserving fine-grained spatial and semantic details.
• It employs varied architectures like interleaved latent blocks, autoregressive visual tokens, and latent slots to integrate multimodal evidence efficiently.
• Recent frameworks demonstrate enhanced performance on tasks such as ScienceQA and spatial reasoning, despite trade-offs in transparency and stability.

Searching arXiv for papers on multimodal latent reasoning and related frameworks. Multimodal latent reasoning denotes a family of methods in which intermediate inference is carried out in continuous hidden representations rather than explicit natural-language rationales or external tool outputs. In this setting, the reasoning state may be a continuous thought vector, a sequence of latent visual tokens, a fixed-capacity bottleneck of “reason tokens,” or an interleaved latent block that is refined before final decoding. The shared motivation is that language-bound Chain-of-Thought can be suboptimal for multimodal problems because token-level rationales are not guaranteed to stay grounded in image features, long textual traces impose inference overhead, and discrete words can fail to preserve fine-grained spatial and semantic structure carried by embeddings (Pham et al., 18 Aug 2025, Li et al., 29 Sep 2025, Jeon et al., 4 Feb 2026).

1. Conceptual scope and historical development

A useful historical precursor is the diffusion-based latent-space formulation in “Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in LLMs,” which conditions the reverse denoising process of an image latent on textual features so that visual representations are aligned with “language thoughts” during rationale generation and answer inference (He et al., 2023). In that formulation, multimodal latent reasoning still serves explicit chain-of-thought generation, but the image latent is no longer treated as a fixed off-the-shelf feature.

From 2025 onward, the emphasis shifts from learning better visual features for textual CoT to moving the reasoning process itself into latent space. “Machine Mental Imagery” introduces interleaved trajectories that mix ordinary text tokens with continuous latent visual tokens produced directly from decoder hidden states (Yang et al., 20 Jun 2025). “Multimodal Chain of Continuous Thought” formalizes a continuous reasoning state $c \in \mathbb{R}^D$ that is iteratively refined without generating natural-language tokens until the final decoding step (Pham et al., 18 Aug 2025). “Latent Visual Reasoning” defines autoregressive reasoning directly in the visual embedding space, interleaved with standard discrete text generation, with latent states trained to reconstruct key visual tokens associated with regions of interest (Li et al., 29 Sep 2025).

Subsequent work broadens the paradigm in several directions. Some systems internalize the effects of external tools or edited trajectories into latent embeddings, as in Pearl’s predictive embedding alignment over expert tool-use trajectories (Adhikari et al., 9 Apr 2026). Others use latent reasoning as an inference-time adaptation mechanism, such as DMLR’s confidence-guided optimization of latent think tokens and dynamic visual injection (Liu et al., 14 Dec 2025). Still others recast the problem around retrieval and universal embedding, where PLUME and LaME replace explicit CoT with short latent rollouts or fixed-capacity reason-token bottlenecks inside the embedding pipeline (He et al., 2 Apr 2026, Wu et al., 11 Jun 2026). This suggests that multimodal latent reasoning has become not a single architecture, but a general design pattern for preserving multimodal evidence inside hidden-state computation.

2. Representational forms and reasoning mechanics

Across the literature, the latent carrier varies, but the basic mechanism is recurrent or autoregressive reuse of hidden states. In MCOUT, visual embeddings $e_v \in \mathbb{R}^{S_v \times D}$ and textual embeddings $e_t \in \mathbb{R}^{S_t \times D}$ are interleaved into a joint sequence, the LLM computes hidden states $h$, and the last hidden state initializes the continuous thought: $c^{(0)} = h[\arg\max(m)-1] \in \mathbb{R}^D$. Reasoning then follows an abstract update $c^{(k)} = f(c^{(k-1)}, e_v, e_t)$, with MCOUT-Base taking $c^{(k)} = h_l^{(k-1)}$ and MCOUT-Multi replacing this with multimodal latent attention over the modality embeddings (Pham et al., 18 Aug 2025).

A second family treats latent reasoning as latent visual token generation. LVR uses special control tokens such as <|lvr_start|> and <|lvr_end|> to switch between text mode and latent mode; in latent mode, the model appends last-layer hidden states as next-step embeddings and trains them to reconstruct selected ROI visual tokens with an MSE objective (Li et al., 29 Sep 2025). Mirage likewise recasts the current hidden state as a latent visual token and appends it to the context, but its two-stage supervision first grounds those latents to compressed image embeddings and then relaxes them under text-only supervision so that the latent trajectory aligns with the task objective (Yang et al., 20 Jun 2025). LanteRn extends this into interleaved latent-text alternation, where continuous latent visual embeddings are emitted inside <|lvr_start|> ... <|lvr_end|> blocks and remain available through the transformer’s attention stream (Viveiros et al., 26 Mar 2026).

A third family uses latent blocks or latent slots as explicit internal workspaces. HyLaR interleaves discrete text generation with continuous visual latent representations delimited by <|canvas_start|> and <|canvas_end|>, treating the reasoning trajectory as a hybrid action space in which each step is either a vocabulary token or a continuous latent vector (Cheng et al., 22 Apr 2026). EVA defines continuous Latent_slot tokens, aligned to the visual encoder’s patch embedding dimension, as intermediate visual thoughts interleaved with textual reasoning (Chen et al., 23 Jun 2026). UniVLR goes further by eliminating explicit textual CoT at inference: textual reasoning traces and auxiliary images are rendered into a single visual workspace during training, compressed into visual latent tokens, and then replaced at inference by a short latent-only reasoning phase followed directly by final answer generation (Jiang et al., 12 May 2026).

A fourth pattern constrains latent reasoning by step-wise grounding to external visual encoders. VaLR dynamically generates vision-aligned latent tokens before each CoT step and aligns those intermediate embeddings to frozen vision encoders such as DINOv3, SigLIP, CLIP, and $\pi^3$ (Jeon et al., 4 Feb 2026). HIVE instead injects hierarchical visual cues from a set of ViT layers $\{6,12,18,24\}$ into a loop transformer across recurrent iterations, so that early latent refinements receive global-to-local or coarse-to-fine visual grounding (Zhang et al., 5 Feb 2026). LaViT aligns not only latent visual semantics but also the teacher’s attention trajectories over image regions, compelling the student to autoregressively reconstruct latent visual thoughts before emitting textual tokens (Wu et al., 15 Jan 2026).

3. Training objectives, supervision, and optimization

The dominant supervised recipe couples token-level language modeling with latent alignment. MCOUT optimizes

$$L_{\text{total}} = \sum_{k=1}^{N_t} \mu \cdot L_{\text{aux}}^{(k)} + L_{\text{final}},$$

where $e_v \in \mathbb{R}^{S_v \times D}$0 is the language-model cross-entropy conditioned on the sequence augmented with the $e_v \in \mathbb{R}^{S_v \times D}$1-th thought embedding, and $e_v \in \mathbb{R}^{S_v \times D}$2 is the answer loss (Pham et al., 18 Aug 2025). LVR jointly optimizes text cross-entropy and a visual reconstruction loss

$e_v \in \mathbb{R}^{S_v \times D}$3

with the vision encoder and projector frozen so that latent states reconstruct query-relevant visual tokens in a stable joint semantic space (Li et al., 29 Sep 2025). UniVLR uses a direct latent alignment term combining LN-MSE and cosine similarity between predicted and target visual latents extracted from a rendered canvas (Jiang et al., 12 May 2026).

Several frameworks replace direct reconstruction with higher-level alignment. Pearl employs a JEPA-style predictive embedding loss,

$e_v \in \mathbb{R}^{S_v \times D}$4

where the model predicts a trajectory-level embedding from the original image-question pair and aligns it to a stop-gradient encoding of the full multimodal trajectory (Adhikari et al., 9 Apr 2026). SLVR adds explicit semantic enrichment: region latent hidden states are aligned to encoder ROI features through an $e_v \in \mathbb{R}^{S_v \times D}$5 loss, while a semantic latent at a special token is projected to a 4096-dimensional attribute embedding and trained with MSE against a structured region profile (Xu et al., 19 May 2026). LatentGeo aligns latent construction tokens to an auxiliary constructed diagram through a hybrid cosine-plus-MSE loss, thereby internalizing auxiliary geometric constructions without pixel rendering (Xu et al., 12 Mar 2026).

Reinforcement learning enters once latent segments become part of the action space. LVR adapts GRPO to mixed text-latent rollouts while computing rewards only on text outputs and replaying latent hidden states for consistent teacher forcing during policy updates (Li et al., 29 Sep 2025). LanteRn combines supervised latent grounding with GRPO and latent state replay to align latent visual thoughts with task-level utility (Viveiros et al., 26 Mar 2026). HyLaR formalizes the latent policy on the hypersphere using a von Mises-Fisher distribution and introduces DePO, which applies independent trust-region constraints to textual and latent components together with an exact closed-form vMF KL regularizer (Cheng et al., 22 Apr 2026). EVA isolates an unstable “transition window”—the first five text tokens after a latent slot—and proposes D-GSPO, which computes the RL objective on text only, masks the ratio to explore regions, and imposes stronger KL regularization on those transition-window tokens (Chen et al., 23 Jun 2026).

Test-time optimization is a distinct branch. DMLR freezes model parameters and refines a small set of latent think tokens by maximizing a confidence reward based on truncated entropy over vocabulary distributions at latent positions. At each iteration it samples Gaussian perturbations, updates the latent tokens with a REINFORCE-like estimator, and injects only the most relevant visual patches selected from internal attention maps (Liu et al., 14 Dec 2025). This makes latent reasoning a runtime control problem rather than only a training-time architectural choice.

4. Empirical results, efficiency, and test-time scaling

The empirical record consistently reports gains on perception-intensive reasoning, though the gains depend strongly on how latent grounding is implemented. MCOUT reports improvements on ScienceQA, MMMU, and MMStar, with up to 8.23% accuracy improvement and up to 8.27% BLEU improvement relative to the baseline; on ScienceQA, MCOUT-Base with $e_v \in \mathbb{R}^{S_v \times D}$6 reaches 58.86% accuracy and BLEU 52.31, while MCOUT-Multi with $e_v \in \mathbb{R}^{S_v \times D}$7 gives the best BLEU of 52.60 (Pham et al., 18 Aug 2025). LVR reports 71.7% on MMVP versus 66.7% for the base Qwen2.5-VL, and 81.7% overall on V* with gains concentrated in detail search and relative positioning (Li et al., 29 Sep 2025). Mirage improves spatial reasoning and planning benchmarks by interleaving latent visual tokens with text, reaching 0.89 average accuracy on VSP spatial reasoning under CoT+GRPO and 0.98 average on SAT Synthetic (Yang et al., 20 Jun 2025).

Several systems emphasize efficiency as much as accuracy. IVT-LR reports an average performance increase of 5.45% in accuracy while achieving a speed increase of over 5 times compared to explicit multimodal CoT baselines; on Qwen2-VL, it uses about 10–11 autoregressive steps rather than tens or hundreds (Chen et al., 14 Oct 2025). PLUME reduces reasoning from 403 generated tokens on average in UME-R1 to 8 latent steps, yielding 30.3× faster inference while improving the average score on the 78-task MMEB-v2 benchmark from 60.1 to 61.6 (He et al., 2 Apr 2026). LaME reports 60× faster inference than explicit CoT methods and 2× faster than latent baselines, with throughput comparable to discriminative embedding models while remaining competitive on MMEB-v2 and MRMR (Wu et al., 11 Jun 2026). EVA reports that at 8192×8192 resolution its latency remains near 8.3 seconds while DeepEyes-7B reaches 53.9 seconds, corresponding to an 84.6% time reduction and approximately 6.5× speedup (Chen et al., 23 Jun 2026).

A separate line of evidence concerns test-time scaling. VaLR is explicitly designed to counter visual information dilution in long-context MLLMs by inserting vision-aligned latent tokens before each reasoning step. On VSI-Bench it improves Qwen2.5-VL-7B from 33.0% to 52.9%, and its performance grows monotonically with longer reasoning chains in settings where prior MLLMs degrade (Jeon et al., 4 Feb 2026). HIVE shows a similar effect through recurrent latent refinement: on ScienceQA-Img, the non-recurrent baseline reaches 60.09, the recurrent model without hierarchical cues reaches 89.39, and the recurrent model with hierarchical cues reaches 91.57 at $e_v \in \mathbb{R}^{S_v \times D}$8 iterations (Zhang et al., 5 Feb 2026). This suggests that latent reasoning can be used not only to compress intermediate computation, but also to make additional internal depth useful in multimodal settings.

5. Failure modes, controversies, and interpretability

A recurrent criticism is that latent reasoning improves efficiency at the cost of transparency. MCOUT states this directly as an interpretability trade-off: continuous thoughts are not human-readable, and although they avoid token overhead, they reduce transparency compared to textual CoT (Pham et al., 18 Aug 2025). UniVLR makes a similar trade-off by eliminating inference-time text CoT entirely in favor of a compact visual latent channel (Jiang et al., 12 May 2026). LaME likewise notes that latent reasoning lacks explicit textual traces and is therefore less interpretable than CoT (Wu et al., 11 Jun 2026).

Another central concern is whether latent methods truly preserve visual grounding. MCOUT’s latent distribution analysis reports a significant norm imbalance on ScienceQA before training, with $e_v \in \mathbb{R}^{S_v \times D}$9 versus $e_t \in \mathbb{R}^{S_t \times D}$0 for attention-mixed thought embeddings, and even after normalization the mixed embeddings show low-variance, near-constant statistics with mean approximately 0.0024 and standard deviation approximately 0.0019; mixed standard deviation is approximately 0.1989 versus last-hidden standard deviation approximately 2.2 (Pham et al., 18 Aug 2025). The paper interprets this as modality collapse and connects it to visual attention sinks. Pearl generalizes the critique by arguing that reconstruction-based latent reasoning methods primarily learn embeddings rather than image edits: training–inference mismatch in the number of latent tokens and near-zero or negative correlation between inference latent-token count and accuracy suggest that many such methods do not in fact simulate intermediate edits (Adhikari et al., 9 Apr 2026).

Distillation-based work identifies a different failure mode. LaViT calls this the “Perception Gap”: on 1000 Visual-CoT examples, a student fine-tuned by standard SFT can match teacher text while diverging in attention on visually demanding tokens, with KL divergence between teacher and student attention maps rising from approximately 1.11 on functional tokens to approximately 1.39 on attribute tokens, while hidden-state cosine distances remain nearly constant at approximately 0.52–0.55 (Wu et al., 15 Jan 2026). The implication is not merely that latent reasoning may fail, but that apparent reasoning improvements can mask reliance on language priors unless latent trajectories are explicitly aligned to teacher visual attention.

Optimization instability is another recurring theme. LVR finds that learned latent end tokens and distance-based stopping are unstable, and that Mode Switching Loss collapses to zero LVR usage (Li et al., 29 Sep 2025). EVA attributes policy deviation to the transition window immediately after latent slots and visualizes token-level KL spikes there during RL (Chen et al., 23 Jun 2026). LatentGeo reports that replacing its latent-aware RL with GRPO or GDPO collapses performance toward 26–27%, largely because the model stops emitting latent segments (Xu et al., 12 Mar 2026). These results indicate that hybrid discrete-continuous policies remain fragile, especially at the text-latent boundary.

6. Major variants and future directions

One axis of variation concerns what the latent state is supposed to encode. Some methods focus on perceptual refresh, as in VaLR’s vision-aligned latent checkpoints and HIVE’s hierarchical visual cue injection (Jeon et al., 4 Feb 2026, Zhang et al., 5 Feb 2026). Others target latent visual imagination or internal sketching, such as Mirage’s helper-image priors, LanteRn’s interleaved latent-text blocks, and Latent Sketchpad’s Context-Aware Vision Head plus Sketch Decoder for human-interpretable sketch rendering (Yang et al., 20 Jun 2025, Viveiros et al., 26 Mar 2026, Zhang et al., 28 Oct 2025). Another group internalizes structured external processes: Pearl absorbs expert tool-use trajectories into predictive embeddings, EVA replaces external tool invocation with continuous latent visual states, and LatentGeo internalizes auxiliary geometric constructions in a latent segment conditioned by a symbolic plan (Adhikari et al., 9 Apr 2026, Chen et al., 23 Jun 2026, Xu et al., 12 Mar 2026).

A second axis concerns the scope of the downstream task. Retrieval-oriented systems such as PLUME and LaME move multimodal latent reasoning away from answer generation and toward universal embedding, where the objective is not to verbalize reasoning but to produce a stronger retrieval vector (He et al., 2 Apr 2026, Wu et al., 11 Jun 2026). SLVR narrows the focus to region-centric consistency under semantic variation by enriching latent region representations with attribute supervision and aligning them across multiple queries grounded in the same region (Xu et al., 19 May 2026). DMLR makes latent reasoning a test-time procedure that “looks only when unsure,” optimizing latent think tokens and dynamically injecting patches when confidence indicates a need for visual evidence (Liu et al., 14 Dec 2025).

The future directions named in the literature are relatively consistent. Several papers propose extension to additional modalities such as audio or video by treating spectrogram patches, frame patches, or interleaved multimodal trajectories as latent inputs to the same reasoning loop (Pham et al., 18 Aug 2025, Adhikari et al., 9 Apr 2026, Jeon et al., 4 Feb 2026). Adaptive compute is another recurring target: MCOUT proposes learned halting criteria, DMLR varies patch injection based on confidence, PLUME uses a fixed latent budget but routes it through an anchor-guided mixture of experts, and HIVE already varies recurrent depth through adaptive early exit (Pham et al., 18 Aug 2025, Liu et al., 14 Dec 2025, He et al., 2 Apr 2026, Zhang et al., 5 Feb 2026). A further common direction is stronger alignment or regularization: MCOUT proposes explicit alignment terms such as contrastive consistency between $e_t \in \mathbb{R}^{S_t \times D}$1 and visual patches, Vedas adds spatially coherent visual replay and routing depth scaling after analyzing gradient imbalance, and LaViT advocates aligning latent attention trajectories rather than only outputs or static embeddings (Pham et al., 18 Aug 2025, Han et al., 12 Apr 2026, Wu et al., 15 Jan 2026).

Taken together, these developments indicate that multimodal latent reasoning is becoming a general substrate for multimodal computation: sometimes as continuous thought refinement, sometimes as latent tool internalization, sometimes as region-centric semantic compression, and sometimes as an efficient bottleneck for retrieval. What remains unsettled is not whether latent reasoning can work, but how best to preserve grounding, maintain stability in hybrid action spaces, and recover enough interpretability for domains where explicit rationales are still required.