Interleaved Latent Visual Reasoning

• Interleaved Latent Visual Reasoning (ILVR) is a multimodal framework that alternates between text generation and autonomous latent visual token synthesis.
• It employs methods like policy gradient reinforcement learning, curriculum training, and selective gradient propagation to optimize integrated visual and textual reasoning.
• ILVR demonstrates significant empirical gains in tasks requiring fine-grained spatial analysis, abstract planning, and efficient multimodal performance.

Interleaved Latent Visual Reasoning (ILVR) denotes a class of multimodal inference frameworks in which Multimodal LLMs (MLLMs) alternate—i.e., interleave—textual chain-of-thought steps and steps manipulating internal continuous latent visual representations. Unlike conventional reasoning methods that either restrict visual semantics to static preconditions or rely on externally tool-driven manipulations, ILVR explicitly constructs, updates, and reasons over intermediate latent visual states within the model’s hidden space. The approach encompasses direct alignment to ground-truth visual features, policy gradient reinforcement learning, and curriculum training protocols to enable autonomous, context-aware generation of visual cues tightly integrated in complex reasoning trajectories. For tasks requiring fine-grained perception, dynamic state evolution, or abstract visual planning, ILVR has demonstrated pronounced empirical and efficiency gains.

1. Conceptual Foundation

ILVR advances beyond conventional multimodal reasoning methods by systematically coupling autoregressive text generation with stepwise synthesis and refinement of latent visual tokens. These latent tokens are derived from frozen or trainable vision encoders (e.g., ViT) and projected into the transformer’s hidden space. At designated positions—either algorithmically triggered by special mode-switch tokens (“<lvr_start>”, “<latent_start>”, or model-internal policy)—the model halts text generation and autonomously produces $K$ continuous latent embeddings that encode current visual semantics. These embeddings can be structured as compressed patch features, context-conditioned edits, or imagined visual thoughts and are attended by subsequent text generation steps. ILVR thus operationalizes an internal “mental imagery” loop: alternating explicit reasoning and latent perception, dynamically updating state-dependent cues and supporting granular, sequential multimodal evidence tracking (Jiang et al., 22 May 2025, Dong et al., 5 Dec 2025, Li et al., 29 Sep 2025).

The approach unifies several architectural and algorithmic strategies:

• Latent token generation via direct reuse of the LLM hidden states (as in “Latent Visual Reasoning” (Li et al., 29 Sep 2025), Mirage (Yang et al., 20 Jun 2025), DeepSketcher (Zhang et al., 30 Sep 2025)),
• Explicit policy optimization on when and how to emit visual latents, either via region selection (R-GRPO (Jiang et al., 22 May 2025)) or visual-latent policy optimization (VLPO (Wang et al., 26 Nov 2025)),
• Auxiliary modules for internal visual editing and sketchpad reasoning (Embedding Editor (Zhang et al., 30 Sep 2025), Latent Sketchpad (Zhang et al., 28 Oct 2025)),
• Self-supervising alignment of latent features via teacher-distilled feature selection (Dong et al., 5 Dec 2025),
• Multi-stage curriculum learning and staged transition from visual supervision to latent autonomy (Wang et al., 26 Nov 2025, Chen et al., 14 Oct 2025).

A plausible implication is that ILVR enables MLLMs to go beyond static visual context and perform dynamic, abstract, or hypothetical visual reasoning—approximating aspects of human visual cognition.

2. Architectural Mechanisms

ILVR frameworks share several core components integrated in distinct architectural variants:

• Vision Encoder & Projection: Encodes input images and/or sub-images into $M$ visual-token embeddings, which are linearly projected to the transformer hidden size and concatenated with text tokens (Li et al., 29 Sep 2025, Wang et al., 26 Nov 2025).
• Latent Token Generation: Upon a mode-switch (e.g., “<latent>”, region tool call, or predicted Crop action), the model generates a contiguous block of $K$ latent visual tokens, either by autoregressively feeding back its own hidden states or through a dedicated head (e.g., Context-Aware Vision Head (Zhang et al., 28 Oct 2025), Embedding Editor (Zhang et al., 30 Sep 2025)). These tokens encode region-specific, imagined, or dynamically evolving representations.
• Integration with Chain-of-Thought: Text decoding resumes, attending to newly appended latent embeddings, enabling subsequent steps to integrate visual evidence created in latent space (Dong et al., 5 Dec 2025, Chen et al., 14 Oct 2025).
• Auxiliary Modules: Models such as DeepSketcher inject an Embedding Editor for tool-free manipulation; Latent Sketchpad uses a Vision Head plus a Sketch Decoder (AlignerNet + VAE) to render human-interpretable images from latents when required (Zhang et al., 30 Sep 2025, Zhang et al., 28 Oct 2025).
• Selective Attention & Masking: Structures attention so that latent tokens attend only to relevant image embeddings or previously generated latents, text tokens attend to latents but not image embeddings, enforcing information flow and grounding (Wang et al., 26 Nov 2025).

The table below categorizes ILVR variants by architectural signature:

System	Visual Latent Mechanism	Auxiliary Modules
VLM-R$^3$ (Jiang et al., 22 May 2025)	Region Crop/Zoom, encoded tokens	R-GRPO, dynamic region policy
DeepSketcher (Zhang et al., 30 Sep 2025)	Embedding Editor latent updates	Cross-attention Q-Former
Latent Sketchpad (Zhang et al., 28 Oct 2025)	Vision Head with contextual cross-attn	Sketch Decoder (AlignerNet+VAE)
Monet (Wang et al., 26 Nov 2025)	Autoregressive latent blocks	Layerwise latent-policy RL
Mirage (Yang et al., 20 Jun 2025)	Direct hidden-state reuse	None
IVT-LR (Chen et al., 14 Oct 2025)	Progressive latent-text+vision	Attention-based fusion

This suggests major variants converge toward modular latent-token generation, context-dependent fusion, and selective attention masking.

3. Training Paradigms and Objectives

ILVR methods employ multi-stage training pipelines combining supervised alignment, curriculum relaxation, and reinforcement learning:

• Supervised Latent Alignment: In initial stages, latent tokens are trained to reconstruct ground-truth visual features—commonly selected helper-image patch embeddings or code-rendered image tokens—using cosine similarity or $L_1$ losses. For instance, DeepSketcher utilizes $$\mathcal{L}_{\mathrm{edit}} = \|\,E_V^{\mathrm{pred}} - E_V^{\mathrm{gt}}\|_{1}$$ (Zhang et al., 30 Sep 2025). Mirage and Monet employ cosine-alignment between generated latents and teacher tokens (Yang et al., 20 Jun 2025, Wang et al., 26 Nov 2025).
• Curriculum Latent Relaxation: After establishing alignment, the model is fine-tuned to generate latent blocks autonomously, relying solely on text-answer cross-entropy (removing visual supervision). This tightens task-specific utility and detaches reasoning from explicit image input (Dong et al., 5 Dec 2025, Li et al., 29 Sep 2025).
• Policy Gradient Reinforcement Learning: RL phases reward correct final answers and format adherence, while direct policy objectives target latent actions (GRPO, VLPO). Monet formulates latent-policy optimization with Gaussian surrogate likelihoods for continuous latents, yielding gradients through hidden states (Wang et al., 26 Nov 2025).
• Selective Backpropagation: Gradients typically flow only through latent-token pathways (latent-only backprop), preventing spurious information leak or collapse of non-latent channels (Wang et al., 26 Nov 2025).

Representative mathematical objectives from Monet (Wang et al., 26 Nov 2025):

$$\mathcal{L}_{\mathrm{align\text{-}obs}} = \frac{1}{N}\sum_{i=1}^N\sum_{\ell=1}^L\Bigl(1 - \cos\bigl(h^{*(i,\ell)}_{\mathrm{obs}},\;\hat h^{(i,\ell)}_{\mathrm{obs}}\bigr)\Bigr)$$

$$\mathcal{L}_{\mathrm{NTP}} = -\frac{1}{N}\sum_{i=1}^N\sum_{t=1}^{T^{(i)}}\log\,p\bigl(y_t\mid y_{<t},\{h\}\bigr)$$

$J(\theta) = \mathbb{E}_{\pi_\theta}\bigl[R\bigr]$

A plausible implication is that dual-signal supervised alignment, RL reward sparsity, and strict gradient masking are essential for robust latent reasoning.

4. Region/Fusion Strategies

Fusion of latent visual reasoning and textual chain-of-thought is accomplished via attention-based integration and structured token streams:

• Region-Conditioned Latent Embedding: VLM-R$^3$ explicitly emits region Crop commands, computes zoom factors based on bounding box area, re-encodes these regions, and appends their embeddings mid-sequence (Jiang et al., 22 May 2025).
• Dynamic Contextual Fusion: Latent steps in IVT-LR (Chen et al., 14 Oct 2025) interleave attention-selected image features with previous hidden states and rationale history, forming fused states passed into the transformer.
• Internal Visual Manipulation: DeepSketcher’s Embedding Editor acts on instruction tokens and current visual state, using a stacked cross-attention block to update latents, which are then directly injected into subsequent reasoning (Zhang et al., 30 Sep 2025).
• Scratchpad and Visual Sketching: Latent Sketchpad maintains an internal visual block updated at “<start_of_image>” triggers. Its Vision Head generates latent visual tokens, which are later decoded into sketches for interpretability, but are always maintained in latent space during core reasoning (Zhang et al., 28 Oct 2025).

This suggests a spectrum of fusion mechanisms: explicit region cropping, self-attended latent loops, context-aware embedding updates, and sketchable visual reasoning.

5. Datasets and Supervision Signals

High-quality supervision is critical for ILVR model performance and generalization:

• Step-Level VLIR Corpus: VLM-R$^3$ employs 11,810 handcrafted interleaved rationales, balancing region selection (Crop) and textual justification, generated via large MLLMs and filtered for semantic and logical coherence (Jiang et al., 22 May 2025).
• DeepSketcher Dataset: 31,000 interleaved CoT trajectories, sourced and expanded via code-rendering and img2code conversion, ensure perfect region fidelity and rich diversity of manipulations (Zhang et al., 30 Sep 2025).
• Monet-SFT-125K Dataset: Construction involves selective retention of unsolved originals, auxiliary image correctness validation, and stringent LLM judging of observation relevance, ensuring only necessary and correct intermediate images are masked and tagged (Wang et al., 26 Nov 2025).
• MazePlanning and Synthetic Benchmarks: Employed in Latent Sketchpad for in-distribution and OOD generalization studies (Zhang et al., 28 Oct 2025). Datasets are decomposed into multi-stage action traces matching visual and textual transitions.

The table below summarizes key ILVR datasets:

Dataset	Size	Modality	Unique Signals
VLIR (Jiang et al., 22 May 2025)	11,810	Interleaved CoT	Crop, zoom, step-wise text
DeepSketcher (Zhang et al., 30 Sep 2025)	31,000	CoT + code edits	Latent edit instructions
Monet-SFT-125K (Wang et al., 26 Nov 2025)	125,000	Image-text CoT	Observation-token tags
MazePlanning (Zhang et al., 28 Oct 2025)	47,800	Mazes, traces	Sketchpad latents

A plausible implication is that agentic curation, high-fidelity region labeling, and explicit observation tagging bridge the supervision gap for latent reasoning.

6. Empirical Impact and Task Generalization

ILVR methods have delivered consistent, state-of-the-art accuracy and speedups across diverse multimodal benchmarks:

• Accuracy Gains: Monet-7B outperforms the Qwen2.5-VL-7B baseline by 6.81 pp (V*), 4.25 pp (HRBench8K), and up to 9.75 pp on MME-RealWorld-Lite (Wang et al., 26 Nov 2025). VLM-R$^3$ yields gains of 2.2–14.3 pp across MathVista, ScienceQA, DocVQA, etc., with the greatest improvements on benchmarks requiring diagrammatic reasoning and spatial cue extraction (Jiang et al., 22 May 2025).
• Efficiency: IVT-LR reduces autoregressive steps by ∼18× and wall-clock latency by 3–8× over state-of-the-art chain-of-focus methods (Chen et al., 14 Oct 2025).
• Generalization: Monet’s VLPO, Mirage, and Latent Sketchpad models retain OOD performance under increased latent block sizes, whereas RL variants targeting only text or format break down (Wang et al., 26 Nov 2025, Zhang et al., 28 Oct 2025).
• Ablations: Removing interleaving, adaptive selection, latent-only gradient, or observation alignment results in drastic accuracy erosion (e.g., Monet V* drops from 83% to 46% without latent-only backprop, DeepSketcher loses up to 12.4 points against tool-based models (Wang et al., 26 Nov 2025, Zhang et al., 30 Sep 2025)).
• Qualitative Analyses: Difference-map visualizations confirm the spatial and semantic accuracy of learned latents: they track the agent’s dynamic path, object transitions, and intentional region edits (Dong et al., 5 Dec 2025, Zhang et al., 30 Sep 2025).

The table below details representative empirical improvements:

Method	Accuracy Gain	Efficiency Gain	Key Task Domains
VLM-R$^3$ (Jiang et al., 22 May 2025)	+2–14 pp	–	Math, ScienceQA, OCR
Monet-7B (Wang et al., 26 Nov 2025)	+6–9 pp	–	Real-world, charts
IVT-LR (Chen et al., 14 Oct 2025)	+5.45 pp	3–8× faster	M³CoT, ScienceQA
DeepSketcher (Zhang et al., 30 Sep 2025)	+3.9–12.4 pp	–	Geometry, logic
Mirage (Yang et al., 20 Jun 2025)	+2–5 pp	–	Spatial planning

This strongly supports the conclusion that ILVR mechanisms are most beneficial for tasks with sequential dynamics, fine-grained spatial reasoning, and hypothetical state transitions.

7. Practical Guidelines and Limitations

Recent research converges on several best practices for ILVR deployment in MLLMs:

1. Dual-Signal Supervision: Combine latent-token alignment with controlled attention to auxiliary image embeddings, enforced via structured attention masks (Wang et al., 26 Nov 2025).
2. Latent-Only Gradient Propagation: Block gradients everywhere except through latent tokens to prevent collapse or cheating (Wang et al., 26 Nov 2025).
3. Layerwise Distillation: Align all transformer layer latents for robust feature propagation.
4. Fixed-Length Latent Blocks: Empirically, 8–12 latent steps balance fidelity and efficiency; larger blocks improve in-distribution accuracy but only latent-specific RL sustains OOD generalization.
5. Explicit RL for Latent Actions: Use Gaussian surrogate likelihoods for latent tokens to enable policy-gradient updates (VLPO). Avoid rewarding mere latent presence, letting the model discover when visual thinking is necessary (Wang et al., 26 Nov 2025).
6. Sparse Reward Design: Only final answer correctness and format should be incentivized.

Limitations include the increased complexity of multi-stage training, fixed overhead per latent step (with minor efficiency penalties), and the need for large, curated interleaved datasets for robust supervision (Dong et al., 5 Dec 2025). Ablation studies warn that omitting any core step dramatically degrades accuracy and adaptability.

• VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (Jiang et al., 22 May 2025)
• DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning (Zhang et al., 30 Sep 2025)
• Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs (Zhang et al., 28 Oct 2025)
• Monet: Reasoning in Latent Visual Space Beyond Images and Language (Wang et al., 26 Nov 2025)
• Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space (Chen et al., 14 Oct 2025)
• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (Yang et al., 20 Jun 2025)
• Interleaved Latent Visual Reasoning with Selective Perceptual Modeling (Dong et al., 5 Dec 2025)
• Latent Visual Reasoning (Li et al., 29 Sep 2025)