---
title: Interleaved Multimodal Chain-of-Thought (iMCoT)
url: https://www.emergentmind.com/topics/interleaved-multimodal-chain-of-thought-imcot
type: topic
---

# Interleaved Multimodal Chain-of-Thought (iMCoT)

Interleaved Multimodal Chain-of-Thought (iMCoT) denotes a class of multimodal reasoning paradigms in which a model alternates textual deliberation with explicit access to visual evidence during the same inference trajectory, rather than encoding the visual input once and then reasoning only in language. Across recent work, the interleaved element may be a selected image region, a grounded visual feature block, a rendered visual thought, a latent visual sketch, an edited or generated image, a cited video frame, or an audio-text stage; the shared aim is to prevent “vision-blind reasoning,” maintain cross-modal consistency, and keep intermediate claims anchored to perceptual state [2606.08464][2411.19488][2606.22565].

## 1. Conceptual foundations

The broader multimodal CoT literature treats chain-of-thought as a generalization of standard CoT to settings where multimodal information can appear in the prompt, query, answer, or rationale. A useful distinction in the survey literature is between text-only rationales and multimodal rationales: the former keep the reasoning trace linguistic, whereas the latter allow non-textual intermediate states to appear inside the chain itself. The exact acronym “iMCoT” is not universal across papers, but the closest recurring notions are step-wise cross-modal reasoning, asynchronous modality modeling, multimodal rationale construction, and repeated perception–reasoning loops [2503.12605].

A central conceptual claim in this line of work is that multimodal reasoning gains are not reducible to merely “adding more images” to prompts. The “Visual Thoughts” perspective argues that both textual-MCoT and interleaved-MCoT work because they introduce “visual thoughts,” a compact intermediate cache of task-relevant visual information that deeper transformer layers can query more effectively than the raw image alone. In that account, performance correlates strongly with clarity and conciseness of the visual-thought representation, while fidelity to the original image is not itself predictive; image-form interleaving tends to help more on fine-grained attributes and operations, whereas textual visual thoughts can suffice for coarse perception [2505.15510].

Recent evaluation work also reframes the motivation for iMCoT as a response to a specific failure mode of multimodal CoT: long verbal reasoning chains often increase “verbal reflection” while visual reflection steadily declines, yielding a “Look Light, Think Heavy” pattern. On 12 multimodal tasks, CoT was beneficial for mathematical, scientific, and multi-image reasoning, but could degrade visual grounding, knowledge-based VQA, and object counting. This makes iMCoT less a generic prompt format than an attempt to maintain visual introspection throughout reasoning rather than only at the input boundary [2606.22565].

## 2. Mechanisms of interleaving

Recent systems instantiate iMCoT through several distinct control primitives.

| Mechanism family | Representative systems | Interleaving primitive |
|---|---|---|
| Attention- or feature-mediated revisit | TVI-CoT, AIMCoT, DaP-ICoT | `<LOOK>` steps, entropy/confidence triggers, top-\(k\) visual tokens, object-centric crops |
| Tool- and crop-mediated grounding | ICoT, VLM-R$^3$, OmniDrive-R1, Simple o3 | attention-selected patches, JSON bboxes, zoom-in calls, `<function>` tool execution |
| Unified text-image or latent generation | ThinkMorph, modal-mixed CoT, Uni-CoT, UniT | image-thought tokens, latent sketches, iterative verify-edit loops |
| Structured temporal or multi-image interleaving | VTI-CoT, CMMCoT | frame–rationale pairs, `<IMG>` and ROI tokens, memory-augmented entity retrieval |

The earliest explicit formulation in this set is ICoT, which writes a reasoning chain as alternating textual rationales and visual patches, \(r_1, x_{v_1}, r_2, x_{v_2}, \ldots\), and realizes the visual insertions through Attention-driven Selection (ADS). ADS uses the decoder’s own text-to-vision attention at a signal token such as a newline, selects the Top-\(K\) visual tokens, restores their spatial order, and inserts them into the autoregressive stream without additional training [2411.19488].

Later systems move from fixed insertion triggers toward learned or adaptive control. TVI-CoT adds learnable control tokens `<THINK>`, `<LOOK>`, and `<ANSWER>`, so the decoder itself decides whether to keep reasoning in text or re-access visual features. At a `<LOOK>` step, it computes reasoning-conditioned attention over visual tokens,
$$
\alpha_t = \mathrm{softmax}\!\left(\frac{(h_t W_q)(v W_k)^\top}{\sqrt{d}}\right),
$$
forms an attended visual representation \(v_t^{att}\), and injects it back into context as an implicit visual token [2606.08464]. DaP-ICoT makes the trigger confidence-based: after a textual step \(T_t\), it computes a confidence score \(C_t\) from token-logit margins and inserts a visual thought only if \(C_t < \tau\), while selecting object-level crops with SAM2 and attention-based relevance scoring [2603.21754]. AIMCoT likewise treats interleaving as an active information-foraging problem: CAG improves attention-map reliability, AVP selects regions by entropy reduction \(IG(\{R_i\}) = U_B - U_{C,i}\), and DAT triggers insertion when text-to-vision attention shifts exceed a threshold \(\delta\) [2509.25699].

Crop-based frameworks make the grounding action itself first-class. OmniDrive-R1 interleaves textual thought tokens with a zoom-in tool call that emits a bounding box \(b_t\) and semantic label \(l_t\), producing a new crop \(I_{t+1} = \mathrm{crop}(I_0, b_t)\) and appending it to the state
$$
s_t = \{I_{\le t}, T_{\le t}\},
$$
so later reasoning is grounded in the original multi-camera scene and all prior inspected views [2512.14044]. VLM-R$^3$ uses JSON crop commands of the form `{"bbox_2d": [x1, y1, x2, y2]}`; once the controller detects a valid crop, it executes crop-plus-zoom, re-encodes the sub-image into visual tokens, and resumes reasoning with the updated state [2505.16192]. Simple o3 generalizes this into an observe–reason–act loop with three explicit tools—`focus_area`, `zoom_in`, and `reuse`—serialized inside `<function>` tags and followed by `<observation>` image tokens [2508.12109].

A separate branch of work abandons explicit retrieval tools and instead lets the model generate or decode internal visual states. ThinkMorph is a unified autoregressive model that alternates textual thoughts and image-thoughts inside a single stream delimited by `<image_start>` and `<image_end>`, with image-thoughts implemented as task-specific visual manipulations such as path overlays, bounding boxes, or jigsaw assemblies [2510.27492]. Modal-mixed CoT uses compact visual sketches represented as latent embeddings between special `<START>` and `<END>` tokens, with a diffusion-based latent decoder conditioned on LLM hidden states [2602.00574]. Uni-CoT and UniT extend the same logic to unified generation and editing: textual plans and visual state updates are interleaved across macro- and micro-level reasoning or across budget-forced iterative edit rounds [2508.05606][2602.12279].

## 3. Data, supervision, and optimization

Training paradigms for iMCoT range from purely inference-time methods to fully supervised and reinforcement-driven pipelines. At one end, ICoT is entirely training-free and parameter-free: ADS works directly on the base model’s attention maps, and AIMCoT is likewise training-free and model-agnostic at inference time [2411.19488][2509.25699]. These methods treat interleaving primarily as a test-time control strategy over existing VLM internals.

Most later work instead constructs explicit interleaved supervision. TVI-CoT trains on approximately 150K interleaved samples synthesized and refined from LLaVA-178k, Visual-CoT, and Zebra-CoT, with supervised next-token prediction plus an auxiliary grounding loss
$$
L = L_{\mathrm{SFT}} + \lambda L_{\mathrm{ground}}, \quad \lambda = 0.1,
$$
where \(L_{\mathrm{ground}}\) concentrates visual attention on annotated or weakly supplied grounding regions [2606.08464]. ThinkMorph is fine-tuned on 24,990 high-quality interleaved traces spanning jigsaw assembly, spatial navigation, visual search, and chart refocus, optimizing text NLL and image-token MSE in a single autoregressive stream [2510.27492]. VTI-CoT automatically generates temporally grounded reasoning trajectories for videos, then renders each step’s rationale and supporting frame onto a single canvas so that the vision encoder can consume a compressed optical version of the full interleaved chain [2606.05736]. Simple o3 uses a verified observe–reason–act synthesis loop to build TWI-Tools-146K, then applies masked SFT over `<reasoning>`, `<function>`, and `<answer>` tokens while treating visual tokens as contextual only [2508.12109]. CMMCoT constructs a 260K multi-image dataset in which CoT traces explicitly emit image indices, ROI coordinates, and optional entity-image segments [2503.05255].

Reinforcement learning is used when the core problem is not only how to represent interleaving, but when to invoke it and how to optimize long-horizon grounding policies. OmniDrive-R1 introduces a pure two-stage RL pipeline with Clip-GRPO. In stage 1, the model learns tool use on 14,452 DeepEyes Q&A pairs with a process-based grounding reward
$$
R_p(\tau) = \sum_{t=1}^{E} \mathrm{sim}_t \lambda^{t-1},
$$
where \(\mathrm{sim}_t\) is CLIP cosine similarity between crop \(I_t\) and generated label \(l_t\); in stage 2, it adapts to 18,507 DriveLMM-o1 examples using answer accuracy and formatting rewards, without dense localization labels [2512.14044]. VLM-R$^3$ similarly combines VLIR supervised fine-tuning with Region-Conditioned Reinforcement Policy Optimization, rewarding exact answers, valid crop syntax, non-redundant region commands, and substantive reasoning text [2505.16192]. Modal-mixed CoT uses SFT on interleaved text-latent traces and then GRPO to teach the policy when to emit visual latents and how to compose longer chains, while UniT relies on next-token prediction over interleaved multimodal trajectories and uses “budget forcing” to scale the number of reasoning rounds at test time [2602.00574][2602.12279].

A recurring supervision theme is the replacement of dense, step-level manual grounding labels with cheaper proxies: CLIP similarity in OmniDrive-R1, weak detector or pretrained grounding targets in TVI-CoT, synthetic but verified tool traces in Simple o3, and rendered optical compression in VTI-CoT. This suggests a shift from annotation-heavy grounding toward process-level or verifier-mediated signals.

## 4. Empirical findings

Reported results show that iMCoT can produce large gains, but also that gains are contingent on accurate grounding and adaptive control rather than the mere presence of longer multimodal traces. In autonomous driving, OmniDrive-R1 improves DriveLMM-o1 overall reasoning from \(51.77\%\) to \(80.35\%\) and final answer accuracy from \(37.81\%\) to \(73.62\%\) relative to the Qwen2.5VL-7B baseline; an ablation without the grounding reward drops to \(70.18\%\) reasoning and \(57.19\%\) MCQ, indicating that process-level visual grounding is not a cosmetic addition but a dominant factor [2512.14044].

On broad multimodal benchmarks, TVI-CoT reports \(64.3\%\) on MMMU, \(60.2\%\) on MathVerse, \(76.6\%\) on MathVista, and \(95.2\%\) on ScienceQA with a Qwen3-VL-8B backbone, corresponding to gains of \(+6.1\), \(+3.8\), \(+3.4\), and \(+3.4\) over the backbone. Its fixed-step ablation shows that adaptive interleaving outperforms any fixed schedule: on MMMU and MathVerse, adaptive control reaches \(64.3/60.2\), whereas forcing 0, 1, 2, 3, or 4 looks never exceeds that combination [2606.08464]. AIMCoT improves both Chameleon-7B and Qwen2-VL-7B-Instruct across M3CoT, ScienceQA, and LLaVA-W, and its ablations show measurable losses when removing CAG, AVP, or DAT, confirming that region choice and timing both matter [2509.25699].

Efficiency-oriented variants demonstrate that better interleaving need not imply more visual insertions. DaP-ICoT reports a \(72.6\%\) reduction in token consumption on M\(^3\)CoT with Qwen2-VL-7B—314 tokens on average versus 1,146 for static ICoT—while still improving M\(^3\)CoT, ScienceQA, and MME; removing DVTI or PVTG degrades performance by \(14.4\%\) and \(13.8\%\) on M\(^3\)CoT, respectively [2603.21754]. ThinkMorph reports an average gain of \(34.74\%\) over the base model on its vision-centric task set and shows that interleaved multimodal thoughts outperform both text-only and visual-only traces on tasks such as jigsaw assembly and visual search [2510.27492]. In video reasoning, VTI-CoT reaches \(65.9\) on MVBench, \(74.5\) on TempCompass, \(59.6\) on Video-MME, \(65.3\) on MMVU, \(55.0\) on LongVideoBench, and \(40.5\) on LVBench, with rendered CoT outperforming tokenized CoT and ordered interleaving outperforming shuffled image–text pairs [2606.05736].

At the same time, recent meta-analysis tempers universalist readings of these results. Across 12 multimodal tasks, multimodal CoT improves mathematical reasoning by \(+6.1\%\), scientific reasoning by \(+2.9\%\), and multi-image reasoning by \(+4.9\%\), but degrades visual grounding by \(-4.6\%\), knowledge-based VQA by \(-3.3\%\), and object counting by \(-4.8\%\). This reinforces a central empirical conclusion of the field: interleaving helps when it restores or preserves visual introspection, but can hurt when it turns perception-heavy tasks into unnecessarily verbose verbal ones [2606.22565].

## 5. Cross-domain extensions

Although image question answering and visual reasoning remain the dominant testbed, iMCoT has already diversified into several application regimes. Autonomous driving is among the most explicit. OmniDrive-R1 starts from six onboard views, lets the policy emit zoom-in actions over the original multi-camera image, and uses the acquired crops to support downstream reasoning about traffic signals, pedestrians, risk assessment, and rule adherence. A qualitative example in the paper shows the model first zooming to confirm a red traffic light and then zooming to verify a pedestrian near a crosswalk before concluding, “Do not proceed; stop and yield” [2512.14044].

Video reasoning extends the same principle temporally. VTI-CoT first segments a video into intervals using CLIP feature-similarity drops, generates step-wise rationales each tagged with a cited interval \([i_{ts}, i_{te}]\), retrieves a representative frame \(V_t\) for each step, and renders the sequence \(\{(r_t,V_t)\}_{t=1}^T\) plus a global summary into a single canvas. This makes temporal grounding explicit and lets the model repeatedly inspect temporally localized evidence rather than rely on a compressed textual summary of the full clip [2606.05736].

Multi-image reasoning places different stress on iMCoT, because the problem is not only revisiting a single scene but matching entities across scenes. CMMCoT interleaves textual rationale fragments with image index tokens such as `<IMG>j</IMG>`, normalized bounding-box coordinates, and optional entity-image segments. At inference, its RIFREM module stores multi-image key–value memories across selected decoder layers and lets entity crops retrieve cross-image evidence through attention, effectively turning multi-image comparison into an interleaved chain of grounded ROI references and memory-augmented feature matching [2503.05255].

The paradigm also generalizes beyond vision in the narrow sense. In end-to-end spoken dialogue, CoT is instantiated as a single decoder-only multimodal LM that interleaves audio and text across three sequential stages—ASR transcript generation, text response generation, and TTS synthesis—using task tokens such as \(\langle asr\rangle\), \(\langle t2t\rangle\), and \(\langle tts\rangle\). This is not crop-based iMCoT, but it is an interleaved multimodal CoT in which audio tokens and text tokens remain within one unified autoregressive stream [2506.00722]. In 3D vision-language learning, structured CoT supervision improves 3D alignment and suggests how explicit multimodal interleaving might be adapted to point-cloud reasoning, although the paper itself notes that there is no strict step-by-step insertion of 3D tokens into the generated text [2503.06232].

A further extension collapses the distinction between understanding and generation. Uni-CoT uses a unified BAGEL backbone to interleave macro-level task planning with micro-level iterative text-image state transitions for generation and editing, while UniT treats multimodal chain-of-thought as a test-time scaling procedure: each round writes verification and planning text, generates or edits the next image, stores both in a shared memory \(M_t=\{X_{\le t}, I_{\le t}\}\), and repeats until the budget is exhausted or the model declares satisfaction [2508.05606][2602.12279]. ThinkMorph pushes the same unification further by making image-thoughts and text-thoughts complementary rather than isomorphic: text plans and verifies, while image steps concretely manipulate or localize content [2510.27492].

## 6. Limitations and open problems

A frequent misconception is that iMCoT is equivalent to “insert more images into the chain.” The literature does not support that view. TVI-CoT shows that fixed schedules underperform adaptive control, and the best number of looks is task-dependent; too many looks can introduce redundancy and noise [2606.08464]. AIMCoT argues that raw attention maps are unreliable and that passive Top-\(K\) region selection misses the model’s actual information need, while its masking experiments show that high-attention regions can be redundant or even misleading [2509.25699]. DaP-ICoT explicitly identifies “Static Visual Thought Positioning” and “Broken Visual Thought Representation” as two major failure modes of earlier interleaved systems [2603.21754].

The strongest cross-paper critique concerns the difficulty of sustaining visual reflection over long chains. The “Look Light, Think Heavy” analysis shows that even explicit reasoning models tend to increase attention to reasoning tokens while progressively neglecting visual tokens, which explains why CoT can hurt grounding and counting even as it helps mathematical or scientific benchmarks [2606.22565]. The preliminary study on inference-time scaling reaches a related conclusion from a different angle: multi-modal thoughts outperform conventional text-only thoughts and blending them increases diversity, but token consumption rises substantially because LOOK steps add operation codes, observations, or additional visual content to the context [2502.11514].

Grounding quality remains another bottleneck. OmniDrive-R1 notes that CLIP alignment may be imperfect for rare classes or adverse conditions, that over-zooming can occur if the decay \(\lambda\) is too weak, and that extreme occlusions or tiny objects remain challenging [2512.14044]. VLM-R$^3$ reports strong sensitivity to bounding-box accuracy and points out that its RL reward directly encourages valid crop syntax and reasoning length, but does not score grounding fidelity beyond those proxies [2505.16192]. Simple o3 depends on accurate tool traces and faces an explicit train–test mismatch because `focus_area` draws a box during training but crops at inference, even though the paper reports that cropping works best at test time [2508.12109].

Representation choice is itself unresolved. The “Visual Thoughts” analysis suggests that the decisive variables are clarity and conciseness rather than whether the intermediate state is textual or image-form, and that different tasks favor different expression types: N-LANG and S-LANG for coarse perception and relational reasoning, E-IMG and G-IMG for fine-grained attributes, operations, and hypothesis externalization [2505.15510]. That finding complicates any attempt to define a single canonical iMCoT format.

The main forward directions are therefore adaptive rather than monolithic. Proposed improvements include adaptive tool-call budgets, learned early stopping, utility-aware or uncertainty-aware penalties for unnecessary looks, stronger domain-adapted grounding backbones, richer tool sets such as depth or BEV queries, temporal memory for long-horizon video reasoning, abstention mechanisms when critical visual evidence is missing, trainable compression for long interleaved traces, and broader evaluation beyond math-heavy RL pipelines [2512.14044][2606.08464][2606.22565]. Taken together, the literature suggests that the most promising future versions of iMCoT will not simply be longer or more image-heavy; they will be systems that decide more reliably when visual re-entry is needed, what visual state should be retrieved or generated, and how that state should remain semantically coupled to the evolving reasoning trace.

Source: https://www.emergentmind.com/topics/interleaved-multimodal-chain-of-thought-imcot