Vision-Interleaved Chain-of-Thought (VICoT)

Updated 13 May 2026

VICoT is a multimodal reasoning framework that interleaves visual states with textual thought to emulate human problem-solving in tasks like spatial planning and remote sensing.
It integrates transformer-based methods with dynamic tool invocation, using visual tokens such as cropped images and latent sketches to optimize reasoning performance.
Empirical studies demonstrate that VICoT consistently outperforms traditional CoT approaches, achieving up to 92.4% accuracy on complex vision-language benchmarks.

Vision-Interleaved Chain-of-Thought (VICoT) is a broad paradigm in multimodal artificial intelligence that fuses stepwise visual feedback with textual reasoning chains. It generalizes classical Chain-of-Thought (CoT) prompting—originating in LLMs—by tightly interleaving image-based intermediate states, cropped views, or visual latent representations with each step of the model’s conceptual reasoning trajectory. This structure is intended to emulate human problem-solving strategies, in which one alternates between looking at visual evidence and “thinking aloud” through multi-step, causally organized decisions. VICoT not only improves performance and interpretability on vision-language (VL) reasoning tasks, but also enables new capabilities such as spatial planning, active visual querying, remote sensing agentics, and context-efficient tool use. The following sections synthesize definitional, methodological, experimental, and architectural aspects of VICoT, distilling key advances across recent literature.

1. Formal Definitions and Theoretical Foundations

VICoT is formally situated as a specialization of multimodal chain-of-thought reasoning. The essential operation is to alternate textual and visual steps in a sequence of intermediate states: $\mathcal{R}_i = \begin{cases} \arg\max_{\mathcal{R}^{\mathcal{E}_{\text{image}}}} \pi_{\text{image}}(\mathbf{R}_{<i},\,\mathcal{V}_I\to\mathcal{R}^{\mathcal{E}_{\text{image}}}) &\text{(visual step)} \ \arg\max_{\mathcal{R}^{\mathcal{E}_{\text{text}}}} \pi_{\text{text}}(\mathbf{R}_{<i},\,\mathcal{V}_I\to\mathcal{R}^{\mathcal{E}_{\text{text}}}) &\text{(textual step)} \end{cases}$ where each $\mathcal{R}_i$ is either a visual “thought” (image, edit, region, or latent) or a textual rationale, with decisions governed by the model’s likelihoods $\pi_{\text{image}}$ and $\pi_{\text{text}}$ (Cheng et al., 21 May 2025). This interleaving forms a trace or trajectory: $\tau = \bigl\{(s_1, I_1),\, (s_2, I_2),\,\dots,\, (s_N, I_N)\bigr\}$ with $s_i$ textual subgoals and $I_i$ associated visual states (keyframes, sketches, region crops) (Liu et al., 1 May 2026).

In vision-language policy models and multi-agent frameworks, VICoT generalizes to alternating between “think” (rationale), “see” (active vision), “act” (tool call) primitives on a recurrent state $S_t$ (Wang et al., 25 Nov 2025, You et al., 2 Feb 2026).

2. Architectural Realizations and Mechanisms

VICoT is realized in multiple architectural variants:

Transformer Interleaving: At each reasoning step, visual thought tokens (derived from cropped images, edit masks, or generated frames) are concatenated with transformer hidden states, enabling the model to carry forward specific visual context:
1. Encode image tokens $\mathbf{X}_{\text{img}}$ to $\mathbf{H}^{(L_i-1)}$ .
2. Generate (or retrieve) intermediate visual tokens $\mathcal{R}_i$ 0.
3. Inject $\mathcal{R}_i$ 1 as additional input to layers $\mathcal{R}_i$ 2 (Cheng et al., 21 May 2025).
Dynamic Tool and Visual Query Integration: Many VICoT systems integrate active modules that select which visual tool to call, or which image region to crop, at each step. For instance, in Simple o3 or VICoT-Agent, the model parses textual reasoning to either execute a tool (e.g., crop, zoom, detect) or proceed with internal reasoning alone, capturing the “observe–reason–act” pattern (Wang et al., 16 Aug 2025, Wang et al., 25 Nov 2025).
Latent Representation Interleaving: In latent VICoT (Shao et al., 31 Jan 2026), compact visual “sketches” (average pooled latent codes, typically via a VLM's vision encoder) are interleaved with text tokens, separated by control tokens (⟨START⟩, ⟨END⟩). A diffusion decoder reconstructs or samples the visual state, enabling lightweight but semantically aligned mental imagery.
Active Perceptual Querying: ViThinker and AIMCoT introduce decision tokens (e.g., <query_depth>) and information-theoretic probes that empower the model to request visual features or spatial patches only when needed, optimizing the trade-off between information gain and computational or token cost (You et al., 2 Feb 2026, Li et al., 30 Sep 2025).
Stack-Based and State-Externalized Reasoning: In agentic frameworks, a stack structure retains tuples (reasoning state, tool call, visual evidence). Each action is followed by a tool-invocation and result update, yielding a trajectory that is both causally ordered and interpretable (Wang et al., 25 Nov 2025). External substrates (e.g., HTML canvas in Canvas-CoT) make visual state mutable and explicitly grounded (Sun et al., 11 Feb 2026).

3. Benchmarking, Evaluation Protocols, and Modalities

Specialized benchmarks and protocols systematically probe VICoT’s impact:

Free-Style Intermediate Visual States (IVS): ViC-Bench evaluates models on tasks (maze navigation, jigsaw puzzle, embodied planning, complex counting) where stepwise actions result in dynamically generated visual states, disallowing static visual contexts (Wu et al., 20 May 2025).
Stagewise Evaluation: Progressive difficulty (multiple-choice, open-ended QA, free-form agent) isolates the effect of interleaved vision on accuracy, correctness (Recall_o), legality (constraint satisfaction), and ThinkGain (change in reasoning accuracy after IVS injection).
Form of Visual Thoughts: Visual thought expressions include Natural Language (captions), Structured Language (scene graphs, JSON), Edited Image (segmentation/annotation), and Generative Image (synthetic steps). Clarity and conciseness are scored and found to strongly correlate with reasoning accuracy (Cheng et al., 21 May 2025).
Robotics and Manipulation: In long-horizon robot planning, VICoT traces ( $\mathcal{R}_i$ 3) combine semantic subgoals (text) and visual keyframes (images/latents), resulting in policies that are both logically and geometrically grounded. Success is measured on manipulation benchmarks such as LIBERO and SimplerEnv-WidowX (Liu et al., 1 May 2026).
Vision-Only, Text-Only, and Interleaved Ablations: Comprehensive ablation shows full interleaved traces outperform text or image alone—e.g., on LIBERO-Long, full VICoT achieves 92.4% vs. 62.0% (text only) (Liu et al., 1 May 2026).

4. Training, Optimization, and Efficiency Considerations

VICoT’s implementation encompasses a diverse landscape of training and inference strategies:

Supervised Finetuning with Interleaved Traces: Annotated multi-modal traces (with alternating reasoning rationales and tool calls/visual tokens) serve as ground truth for SFT, typically with selective loss masking on text vs. vision tokens (Wang et al., 16 Aug 2025).
Reinforcement Learning and Preference Optimization: RL paradigms, especially in safety-critical or robotic settings, employ process-based (e.g., grounding via CLIP similarity) and outcome-based rewards, often via actor-critic, PPO, or Trust-Region methods. Fine-tuning learns both when to use visual tools and how to sequence them, e.g., Clip-GRPO (Zhang et al., 16 Dec 2025).
Inference-Time Routing and Conditional Interleaving: Some frameworks (e.g., DaP-ICoT, AIMCoT) eschew additional training in favor of inference-time mechanisms: logit margin confidence checks or dynamic attention-shift triggers determine when to inject new visual content (Liu et al., 23 Mar 2026, Li et al., 30 Sep 2025).
Token and Computation Budgeting: Token and compute overheads are addressed by ensuring visual states are compact (e.g., latent embeddings, object-level crops), dynamically selected, or sparsity-penalized (Shao et al., 31 Jan 2026, Li et al., 30 Sep 2025, Liu et al., 23 Mar 2026). For example, DaP-ICoT reduces average token use by 72.6% compared to static ICoT (Liu et al., 23 Mar 2026).
Stack Distillation and Edge Deployment: Reasoning stack distillation enables smaller models to learn from large teacher trajectories, reducing memory/compute requirements while maintaining high performance and quality on complex remote sensing tasks (Wang et al., 25 Nov 2025).

5. Empirical Findings and Impact on Multimodal Reasoning

Empirical evidence across diverse tasks converges on the superiority of VICoT architectures:

Model/Framework	Benchmark	Text Only	Vision Only	VICoT (Interleaved)
Show-o2-like Transformer (Liu et al., 1 May 2026)	LIBERO-Long (success %)	62.0	68.4	92.4
Chameleon-7B (Li et al., 30 Sep 2025)	M3CoT ACC (zero-shot)	—	—	31.4
Simple o3 (Wang et al., 16 Aug 2025)	MME (reasoning)	652.5	—	702.1
VICoT (distilled) (Wang et al., 25 Nov 2025)	RSVQA-HR (avg. accuracy %)	75.3	—	92.3

Experiments demonstrate:

Consistent absolute improvements: VICoT delivers +3–8 points accuracy over state-of-the-art text-only CoT baselines on fine-grained, attribute, or multi-step reasoning tasks (Cheng et al., 21 May 2025, Wu et al., 20 May 2025).
Token efficiency vs. accuracy trade-off: More detailed visual thoughts (Edited/Generated images) offer higher accuracy but at the cost of increased tokens—1,100+/sample—but latent/interleaved designs mitigate this (Shao et al., 31 Jan 2026).
Interpretability and robustness: Interleaved traces and stack-based rationales yield highly interpretable and auditable outputs, facilitate error correction, and are robust to small perturbations and missing states (Sun et al., 11 Feb 2026, Liu et al., 1 May 2026).

6. Limitations, Extensions, and Future Directions

Several limitations and open directions emerge:

Error Correction and State Mutability: Classic VICoT’s immutable visual snapshots hamper correction efficiency; substrates such as Canvas-CoT’s structured HTML DOM enable in-place, non-monotonic edits (Sun et al., 11 Feb 2026).
Complex Scene Scalability: High-dimensional/large-scale reasoning (e.g., remote sensing with UHR imagery) still challenges both inference speed and tool selection logic (Wang et al., 25 Nov 2025).
Adaptive Modality and Layer Injection: There is strong evidence for developing adaptive strategies to select whether to inject text or image thoughts, and to tune optimal network layers for fusion (Cheng et al., 21 May 2025).
Annotation Cost and Labeling: VI-CoT performance is sensitive to the quality of visual state supervision; scalable pseudo-supervision (e.g., automatic segmentation, auto-captioning) is an important practical concern (Liu et al., 1 May 2026).
Generalization Across Domains: Although substantial gains are shown in robotics, remote sensing, and driving, further validation is required for open-domain, uncurated tasks (Wang et al., 25 Nov 2025).
Streaming and Real-Time Deployment: Real-time, low-latency VICoT, especially with on-the-fly tool invocation and limited hardware, remains an open engineering challenge.

VICoT is distinct from, but conceptually linked to:

Standard Multimodal CoT (MCoT): Lacks explicit visual state interleaving, operating either as text-only rationales (T-MCoT) or via static, precomputed visual tokens. VICoT (I-MCoT) emerges as the most neural-cognitively aligned (Cheng et al., 21 May 2025).
Passive vs. Active Perceptual Querying: Passive methods enumerate or statically select visual features; active frameworks introduce explicit querying, foraging, or tool invocation based on uncertainty or information gain, yielding stricter grounding and efficiency (You et al., 2 Feb 2026, Li et al., 30 Sep 2025).
Mutable State Architectures: Newer state-externalized architectures (e.g., Canvas-CoT) shift VICoT from linear trace models to non-monotonic, locally revisable state spaces that better reflect iterative, constraint-enforced reasoning (Sun et al., 11 Feb 2026).

In summary, Vision-Interleaved Chain-of-Thought is established as the modern foundation for interpretable, grounded, and efficient multimodal reasoning. By jointly modeling perception and cognition in recursive feedback loops, VICoT unlocks new levels of generalization, task fidelity, and transparency across domains requiring deep visual understanding and active, iterative planning.