Interleaved Multimodal Chain-of-Thought (iMCoT)

• Interleaved Multimodal Chain-of-Thought (iMCoT) is a paradigm that alternates textual reasoning with visual evidence to enhance cross-modal consistency and prevent vision-blind reasoning.
• It employs adaptive interleaving mechanisms—such as attention triggers and tool-mediated crop retrieval—to dynamically fuse visual and linguistic information during inference.
• Empirical evaluations demonstrate that iMCoT improves performance in tasks like autonomous driving and multi-image/video analysis while optimizing token efficiency.

Interleaved Multimodal Chain-of-Thought (iMCoT) denotes a class of multimodal reasoning paradigms in which a model alternates textual deliberation with explicit access to visual evidence during the same inference trajectory, rather than encoding the visual input once and then reasoning only in language. Across recent work, the interleaved element may be a selected image region, a grounded visual feature block, a rendered visual thought, a latent visual sketch, an edited or generated image, a cited video frame, or an audio-text stage; the shared aim is to prevent “vision-blind reasoning,” maintain cross-modal consistency, and keep intermediate claims anchored to perceptual state (Hu et al., 7 Jun 2026, Gao et al., 2024, Jin et al., 21 Jun 2026).

1. Conceptual foundations

The broader multimodal CoT literature treats chain-of-thought as a generalization of standard CoT to settings where multimodal information can appear in the prompt, query, answer, or rationale. A useful distinction in the survey literature is between text-only rationales and multimodal rationales: the former keep the reasoning trace linguistic, whereas the latter allow non-textual intermediate states to appear inside the chain itself. The exact acronym “iMCoT” is not universal across papers, but the closest recurring notions are step-wise cross-modal reasoning, asynchronous modality modeling, multimodal rationale construction, and repeated perception–reasoning loops (Wang et al., 16 Mar 2025).

A central conceptual claim in this line of work is that multimodal reasoning gains are not reducible to merely “adding more images” to prompts. The “Visual Thoughts” perspective argues that both textual-MCoT and interleaved-MCoT work because they introduce “visual thoughts,” a compact intermediate cache of task-relevant visual information that deeper transformer layers can query more effectively than the raw image alone. In that account, performance correlates strongly with clarity and conciseness of the visual-thought representation, while fidelity to the original image is not itself predictive; image-form interleaving tends to help more on fine-grained attributes and operations, whereas textual visual thoughts can suffice for coarse perception (Cheng et al., 21 May 2025).

Recent evaluation work also reframes the motivation for iMCoT as a response to a specific failure mode of multimodal CoT: long verbal reasoning chains often increase “verbal reflection” while visual reflection steadily declines, yielding a “Look Light, Think Heavy” pattern. On 12 multimodal tasks, CoT was beneficial for mathematical, scientific, and multi-image reasoning, but could degrade visual grounding, knowledge-based VQA, and object counting. This makes iMCoT less a generic prompt format than an attempt to maintain visual introspection throughout reasoning rather than only at the input boundary (Jin et al., 21 Jun 2026).

2. Mechanisms of interleaving

Recent systems instantiate iMCoT through several distinct control primitives.

Mechanism family	Representative systems	Interleaving primitive
Attention- or feature-mediated revisit	TVI-CoT, AIMCoT, DaP-ICoT	<LOOK> steps, entropy/confidence triggers, top-$k$ visual tokens, object-centric crops
Tool- and crop-mediated grounding	ICoT, VLM-R$^3$, OmniDrive-R1, Simple o3	attention-selected patches, JSON bboxes, zoom-in calls, <function> tool execution
Unified text-image or latent generation	ThinkMorph, modal-mixed CoT, Uni-CoT, UniT	image-thought tokens, latent sketches, iterative verify-edit loops
Structured temporal or multi-image interleaving	VTI-CoT, CMMCoT	frame–rationale pairs, <IMG> and ROI tokens, memory-augmented entity retrieval

The earliest explicit formulation in this set is ICoT, which writes a reasoning chain as alternating textual rationales and visual patches, $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$, and realizes the visual insertions through Attention-driven Selection (ADS). ADS uses the decoder’s own text-to-vision attention at a signal token such as a newline, selects the Top-$K$ visual tokens, restores their spatial order, and inserts them into the autoregressive stream without additional training (Gao et al., 2024).

Later systems move from fixed insertion triggers toward learned or adaptive control. TVI-CoT adds learnable control tokens <THINK>, <LOOK>, and <ANSWER>, so the decoder itself decides whether to keep reasoning in text or re-access visual features. At a <LOOK> step, it computes reasoning-conditioned attention over visual tokens,

$$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$

forms an attended visual representation $v_t^{att}$, and injects it back into context as an implicit visual token (Hu et al., 7 Jun 2026). DaP-ICoT makes the trigger confidence-based: after a textual step $T_t$, it computes a confidence score $C_t$ from token-logit margins and inserts a visual thought only if $C_t < \tau$, while selecting object-level crops with SAM2 and attention-based relevance scoring (Liu et al., 23 Mar 2026). AIMCoT likewise treats interleaving as an active information-foraging problem: CAG improves attention-map reliability, AVP selects regions by entropy reduction $IG(\{R_i\}) = U_B - U_{C,i}$, and DAT triggers insertion when text-to-vision attention shifts exceed a threshold $^3$0 (Li et al., 30 Sep 2025).

Crop-based frameworks make the grounding action itself first-class. OmniDrive-R1 interleaves textual thought tokens with a zoom-in tool call that emits a bounding box $^3$1 and semantic label $^3$2, producing a new crop $^3$3 and appending it to the state

$^3$4

so later reasoning is grounded in the original multi-camera scene and all prior inspected views (Zhang et al., 16 Dec 2025). VLM-R$^3$5 uses JSON crop commands of the form {"bbox_2d": [x1, y1, x2, y2]}; once the controller detects a valid crop, it executes crop-plus-zoom, re-encodes the sub-image into visual tokens, and resumes reasoning with the updated state (Jiang et al., 22 May 2025). Simple o3 generalizes this into an observe–reason–act loop with three explicit tools—focus_area, zoom_in, and reuse—serialized inside <function> tags and followed by <observation> image tokens (Wang et al., 16 Aug 2025).

A separate branch of work abandons explicit retrieval tools and instead lets the model generate or decode internal visual states. ThinkMorph is a unified autoregressive model that alternates textual thoughts and image-thoughts inside a single stream delimited by <image_start> and <image_end>, with image-thoughts implemented as task-specific visual manipulations such as path overlays, bounding boxes, or jigsaw assemblies (Gu et al., 30 Oct 2025). Modal-mixed CoT uses compact visual sketches represented as latent embeddings between special <START> and <END> tokens, with a diffusion-based latent decoder conditioned on LLM hidden states (Shao et al., 31 Jan 2026). Uni-CoT and UniT extend the same logic to unified generation and editing: textual plans and visual state updates are interleaved across macro- and micro-level reasoning or across budget-forced iterative edit rounds (Qin et al., 7 Aug 2025, Chen et al., 12 Feb 2026).

3. Data, supervision, and optimization

Training paradigms for iMCoT range from purely inference-time methods to fully supervised and reinforcement-driven pipelines. At one end, ICoT is entirely training-free and parameter-free: ADS works directly on the base model’s attention maps, and AIMCoT is likewise training-free and model-agnostic at inference time (Gao et al., 2024, Li et al., 30 Sep 2025). These methods treat interleaving primarily as a test-time control strategy over existing VLM internals.

Most later work instead constructs explicit interleaved supervision. TVI-CoT trains on approximately 150K interleaved samples synthesized and refined from LLaVA-178k, Visual-CoT, and Zebra-CoT, with supervised next-token prediction plus an auxiliary grounding loss

$^3$6

where $^3$7 concentrates visual attention on annotated or weakly supplied grounding regions (Hu et al., 7 Jun 2026). ThinkMorph is fine-tuned on 24,990 high-quality interleaved traces spanning jigsaw assembly, spatial navigation, visual search, and chart refocus, optimizing text NLL and image-token MSE in a single autoregressive stream (Gu et al., 30 Oct 2025). VTI-CoT automatically generates temporally grounded reasoning trajectories for videos, then renders each step’s rationale and supporting frame onto a single canvas so that the vision encoder can consume a compressed optical version of the full interleaved chain (Zhang et al., 4 Jun 2026). Simple o3 uses a verified observe–reason–act synthesis loop to build TWI-Tools-146K, then applies masked SFT over <reasoning>, <function>, and <answer> tokens while treating visual tokens as contextual only (Wang et al., 16 Aug 2025). CMMCoT constructs a 260K multi-image dataset in which CoT traces explicitly emit image indices, ROI coordinates, and optional entity-image segments (Zhang et al., 7 Mar 2025).

Reinforcement learning is used when the core problem is not only how to represent interleaving, but when to invoke it and how to optimize long-horizon grounding policies. OmniDrive-R1 introduces a pure two-stage RL pipeline with Clip-GRPO. In stage 1, the model learns tool use on 14,452 DeepEyes Q&A pairs with a process-based grounding reward

$^3$8

where $^3$9 is CLIP cosine similarity between crop $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$0 and generated label $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$1; in stage 2, it adapts to 18,507 DriveLMM-o1 examples using answer accuracy and formatting rewards, without dense localization labels (Zhang et al., 16 Dec 2025). VLM-R$r_1, x_{v_1}, r_2, x_{v_2}, \ldots$2 similarly combines VLIR supervised fine-tuning with Region-Conditioned Reinforcement Policy Optimization, rewarding exact answers, valid crop syntax, non-redundant region commands, and substantive reasoning text (Jiang et al., 22 May 2025). Modal-mixed CoT uses SFT on interleaved text-latent traces and then GRPO to teach the policy when to emit visual latents and how to compose longer chains, while UniT relies on next-token prediction over interleaved multimodal trajectories and uses “budget forcing” to scale the number of reasoning rounds at test time (Shao et al., 31 Jan 2026, Chen et al., 12 Feb 2026).

A recurring supervision theme is the replacement of dense, step-level manual grounding labels with cheaper proxies: CLIP similarity in OmniDrive-R1, weak detector or pretrained grounding targets in TVI-CoT, synthetic but verified tool traces in Simple o3, and rendered optical compression in VTI-CoT. This suggests a shift from annotation-heavy grounding toward process-level or verifier-mediated signals.

4. Empirical findings

Reported results show that iMCoT can produce large gains, but also that gains are contingent on accurate grounding and adaptive control rather than the mere presence of longer multimodal traces. In autonomous driving, OmniDrive-R1 improves DriveLMM-o1 overall reasoning from $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$3 to $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$4 and final answer accuracy from $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$5 to $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$6 relative to the Qwen2.5VL-7B baseline; an ablation without the grounding reward drops to $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$7 reasoning and $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$8 MCQ, indicating that process-level visual grounding is not a cosmetic addition but a dominant factor (Zhang et al., 16 Dec 2025).

On broad multimodal benchmarks, TVI-CoT reports $r_1, x_{v_1}, r_2, x_{v_2}, \ldots$9 on MMMU, $K$0 on MathVerse, $K$1 on MathVista, and $K$2 on ScienceQA with a Qwen3-VL-8B backbone, corresponding to gains of $K$3, $K$4, $K$5, and $K$6 over the backbone. Its fixed-step ablation shows that adaptive interleaving outperforms any fixed schedule: on MMMU and MathVerse, adaptive control reaches $K$7, whereas forcing 0, 1, 2, 3, or 4 looks never exceeds that combination (Hu et al., 7 Jun 2026). AIMCoT improves both Chameleon-7B and Qwen2-VL-7B-Instruct across M3CoT, ScienceQA, and LLaVA-W, and its ablations show measurable losses when removing CAG, AVP, or DAT, confirming that region choice and timing both matter (Li et al., 30 Sep 2025).

Efficiency-oriented variants demonstrate that better interleaving need not imply more visual insertions. DaP-ICoT reports a $K$8 reduction in token consumption on M$K$9CoT with Qwen2-VL-7B—314 tokens on average versus 1,146 for static ICoT—while still improving M$$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$0CoT, ScienceQA, and MME; removing DVTI or PVTG degrades performance by $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$1 and $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$2 on M$$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$3CoT, respectively (Liu et al., 23 Mar 2026). ThinkMorph reports an average gain of $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$4 over the base model on its vision-centric task set and shows that interleaved multimodal thoughts outperform both text-only and visual-only traces on tasks such as jigsaw assembly and visual search (Gu et al., 30 Oct 2025). In video reasoning, VTI-CoT reaches $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$5 on MVBench, $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$6 on TempCompass, $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$7 on Video-MME, $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$8 on MMVU, $$\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),$$9 on LongVideoBench, and $v_t^{att}$0 on LVBench, with rendered CoT outperforming tokenized CoT and ordered interleaving outperforming shuffled image–text pairs (Zhang et al., 4 Jun 2026).

At the same time, recent meta-analysis tempers universalist readings of these results. Across 12 multimodal tasks, multimodal CoT improves mathematical reasoning by $v_t^{att}$1, scientific reasoning by $v_t^{att}$2, and multi-image reasoning by $v_t^{att}$3, but degrades visual grounding by $v_t^{att}$4, knowledge-based VQA by $v_t^{att}$5, and object counting by $v_t^{att}$6. This reinforces a central empirical conclusion of the field: interleaving helps when it restores or preserves visual introspection, but can hurt when it turns perception-heavy tasks into unnecessarily verbose verbal ones (Jin et al., 21 Jun 2026).

5. Cross-domain extensions

Although image question answering and visual reasoning remain the dominant testbed, iMCoT has already diversified into several application regimes. Autonomous driving is among the most explicit. OmniDrive-R1 starts from six onboard views, lets the policy emit zoom-in actions over the original multi-camera image, and uses the acquired crops to support downstream reasoning about traffic signals, pedestrians, risk assessment, and rule adherence. A qualitative example in the paper shows the model first zooming to confirm a red traffic light and then zooming to verify a pedestrian near a crosswalk before concluding, “Do not proceed; stop and yield” (Zhang et al., 16 Dec 2025).

Video reasoning extends the same principle temporally. VTI-CoT first segments a video into intervals using CLIP feature-similarity drops, generates step-wise rationales each tagged with a cited interval $v_t^{att}$7, retrieves a representative frame $v_t^{att}$8 for each step, and renders the sequence $v_t^{att}$9 plus a global summary into a single canvas. This makes temporal grounding explicit and lets the model repeatedly inspect temporally localized evidence rather than rely on a compressed textual summary of the full clip (Zhang et al., 4 Jun 2026).

Multi-image reasoning places different stress on iMCoT, because the problem is not only revisiting a single scene but matching entities across scenes. CMMCoT interleaves textual rationale fragments with image index tokens such as <IMG>j</IMG>, normalized bounding-box coordinates, and optional entity-image segments. At inference, its RIFREM module stores multi-image key–value memories across selected decoder layers and lets entity crops retrieve cross-image evidence through attention, effectively turning multi-image comparison into an interleaved chain of grounded ROI references and memory-augmented feature matching (Zhang et al., 7 Mar 2025).

The paradigm also generalizes beyond vision in the narrow sense. In end-to-end spoken dialogue, CoT is instantiated as a single decoder-only multimodal LM that interleaves audio and text across three sequential stages—ASR transcript generation, text response generation, and TTS synthesis—using task tokens such as $T_t$0, $T_t$1, and $T_t$2. This is not crop-based iMCoT, but it is an interleaved multimodal CoT in which audio tokens and text tokens remain within one unified autoregressive stream (Arora et al., 31 May 2025). In 3D vision-language learning, structured CoT supervision improves 3D alignment and suggests how explicit multimodal interleaving might be adapted to point-cloud reasoning, although the paper itself notes that there is no strict step-by-step insertion of 3D tokens into the generated text (Chen et al., 8 Mar 2025).

A further extension collapses the distinction between understanding and generation. Uni-CoT uses a unified BAGEL backbone to interleave macro-level task planning with micro-level iterative text-image state transitions for generation and editing, while UniT treats multimodal chain-of-thought as a test-time scaling procedure: each round writes verification and planning text, generates or edits the next image, stores both in a shared memory $T_t$3, and repeats until the budget is exhausted or the model declares satisfaction (Qin et al., 7 Aug 2025, Chen et al., 12 Feb 2026). ThinkMorph pushes the same unification further by making image-thoughts and text-thoughts complementary rather than isomorphic: text plans and verifies, while image steps concretely manipulate or localize content (Gu et al., 30 Oct 2025).

6. Limitations and open problems

A frequent misconception is that iMCoT is equivalent to “insert more images into the chain.” The literature does not support that view. TVI-CoT shows that fixed schedules underperform adaptive control, and the best number of looks is task-dependent; too many looks can introduce redundancy and noise (Hu et al., 7 Jun 2026). AIMCoT argues that raw attention maps are unreliable and that passive Top-$T_t$4 region selection misses the model’s actual information need, while its masking experiments show that high-attention regions can be redundant or even misleading (Li et al., 30 Sep 2025). DaP-ICoT explicitly identifies “Static Visual Thought Positioning” and “Broken Visual Thought Representation” as two major failure modes of earlier interleaved systems (Liu et al., 23 Mar 2026).

The strongest cross-paper critique concerns the difficulty of sustaining visual reflection over long chains. The “Look Light, Think Heavy” analysis shows that even explicit reasoning models tend to increase attention to reasoning tokens while progressively neglecting visual tokens, which explains why CoT can hurt grounding and counting even as it helps mathematical or scientific benchmarks (Jin et al., 21 Jun 2026). The preliminary study on inference-time scaling reaches a related conclusion from a different angle: multi-modal thoughts outperform conventional text-only thoughts and blending them increases diversity, but token consumption rises substantially because LOOK steps add operation codes, observations, or additional visual content to the context (Lin et al., 17 Feb 2025).

Grounding quality remains another bottleneck. OmniDrive-R1 notes that CLIP alignment may be imperfect for rare classes or adverse conditions, that over-zooming can occur if the decay $T_t$5 is too weak, and that extreme occlusions or tiny objects remain challenging (Zhang et al., 16 Dec 2025). VLM-R$T_t$6 reports strong sensitivity to bounding-box accuracy and points out that its RL reward directly encourages valid crop syntax and reasoning length, but does not score grounding fidelity beyond those proxies (Jiang et al., 22 May 2025). Simple o3 depends on accurate tool traces and faces an explicit train–test mismatch because focus_area draws a box during training but crops at inference, even though the paper reports that cropping works best at test time (Wang et al., 16 Aug 2025).

Representation choice is itself unresolved. The “Visual Thoughts” analysis suggests that the decisive variables are clarity and conciseness rather than whether the intermediate state is textual or image-form, and that different tasks favor different expression types: N-LANG and S-LANG for coarse perception and relational reasoning, E-IMG and G-IMG for fine-grained attributes, operations, and hypothesis externalization (Cheng et al., 21 May 2025). That finding complicates any attempt to define a single canonical iMCoT format.

The main forward directions are therefore adaptive rather than monolithic. Proposed improvements include adaptive tool-call budgets, learned early stopping, utility-aware or uncertainty-aware penalties for unnecessary looks, stronger domain-adapted grounding backbones, richer tool sets such as depth or BEV queries, temporal memory for long-horizon video reasoning, abstention mechanisms when critical visual evidence is missing, trainable compression for long interleaved traces, and broader evaluation beyond math-heavy RL pipelines (Zhang et al., 16 Dec 2025, Hu et al., 7 Jun 2026, Jin et al., 21 Jun 2026). Taken together, the literature suggests that the most promising future versions of iMCoT will not simply be longer or more image-heavy; they will be systems that decide more reliably when visual re-entry is needed, what visual state should be retrieved or generated, and how that state should remain semantically coupled to the evolving reasoning trace.