Interleaved Chain-of-Thought Reasoning

• Interleaved CoT reasoning is a multimodal framework that alternates text and visual tokens to create step-by-step, perceptually grounded explanations.
• It utilizes dynamic token interleaving with attention-driven selection and reinforcement techniques to improve cross-modal alignment and performance.
• Empirical results show marked accuracy gains and enhanced interpretability over traditional text-only approaches due to explicit visual grounding.

Interleaved Chain-of-Thought (CoT) Reasoning refers to a multimodal reasoning paradigm in which models generate alternating sequences of textual and visual tokens—such as images, visual crops, sketches, or video frames—so that each step of the reasoning process is explicitly grounded in both modalities. This framework generalizes standard Chain-of-Thought approaches by tightly integrating language and perception at the token or step level, aligning computational reasoning more closely with human problem-solving practices that fluidly combine textual and visual aids.

1. Formal Definition and Multimodal Sequence Structure

Interleaved CoT reasoning yields a sequence

$\mathcal{Y} = (y_1, v_1, y_2, v_2, \dots, y_N, a)$

where each $y_i$ is a segment of text (explanation or substep), each $v_i$ is a visual token (e.g., crop, mask, sketch, diagram, or key-frame), and $a$ is the final answer or output. The elements are produced in an auto-regressive manner from a unified model $f_\theta$ conditioned on the original input (be it image, video, or text prompt). Formally, this process factorizes via

$$P(\mathcal{Y}|X,P) = \prod_{t=1}^T P(y_t | y_{<t}, X, P)$$

with special markers to delimit visual tokens (such as <image_start>...</image_end>, or similar architecture-specific tags). Unlike text-only CoT, interleaved CoT chains are not monolithic: each reasoning step may correspond to direct perception, a visual manipulation, or a symbolic transformation, depending on the current context and learned curriculum (Zhang et al., 14 Jul 2025, Chen et al., 5 Jun 2025, Li et al., 22 Jul 2025, Gu et al., 30 Oct 2025, Zou et al., 9 Oct 2025, Gao et al., 29 Nov 2024, Zhang et al., 16 Dec 2025).

2. Core Mechanisms and Model Implementations

Token Interleaving and Cross-Modal Attentional Updates

Implementations universally rely on joint encoders (or encoder-decoder architectures) supporting alternating modalities. At step $t$, the hidden state is updated as

$h_t = f_\theta(h_{t-1}, x_t)$

where each token $x_t$ is either a text token, an image embedding, a video frame feature, or other visual entity. For example, in video-text setups, the system can seamlessly alternate between token types: $$x_t = \begin{cases} e^{\mathrm{text}}_t & \text{if text} \ e^{\mathrm{vis}}_t & \text{if key-frame, crop, etc.} \end{cases}$$ This interleaving unlocks the ability of the model to ground each text rationale in actual perceptual evidence and to allow for context-dependent visual or textual generation (Zhang et al., 14 Jul 2025, Chen et al., 5 Jun 2025, Gu et al., 30 Oct 2025).

Dynamic Selection and Insertion

Concrete mechanisms for visual token selection include Attention-driven Selection (ADS) (Gao et al., 29 Nov 2024), learned Interleave Token scoring (Chen et al., 5 Jun 2025), and model-internal tool invocation (such as zoom-in with reinforcement-driven localization (Zhang et al., 16 Dec 2025)). These allow for dynamic, context-dependent retrieval or synthesis of relevant visual features during the reasoning process.

The following table summarizes selection paradigms in recent studies:

Method	Visual Selection Mechanism	Token Granularity
ViTCoT (Zhang et al., 14 Jul 2025)	Key-frame extraction + manual/hybrid verification	Short video segments (frames)
MINT-CoT (Chen et al., 5 Jun 2025)	Learnable Interleave Token + cosine threshold	Arbitrary-shaped visual tokens
ICoT (Gao et al., 29 Nov 2024)	Attention-driven Selection (ADS, plug-and-play)	Cross-modal patch attention
OmniDrive-R1 (Zhang et al., 16 Dec 2025)	RL-based zoom-tool invocations (Clip-GRPO)	Bbox-cropped image regions
Zebra-CoT (Li et al., 22 Jul 2025)	Dataset-level, curated human+synthetic traces	Full images, synthetic sketches

3. Training Paradigms and Loss Functions

End-to-end training leverages composite objectives that supervise both the textual and visual reasoning streams. These typically include:

• Cross-entropy loss for text token generation.
• MSE loss for visual tokens (pixels, diffusion latents, or segmentations).
• Alignment objectives (e.g., cosine similarity between visual and language embeddings (Zhang et al., 14 Jul 2025)).
• Reinforcement Learning stages (e.g., Group Relative Policy Optimization (GRPO), process-based grounding rewards via CLIP similarity (Zhang et al., 16 Dec 2025, Chen et al., 5 Jun 2025)).

Total loss may combine several components: $$\mathcal{L} = \lambda_t \mathcal{L}_{\text{text}} + \lambda_v \mathcal{L}_{\text{vision}} + \lambda_{\text{align}} \mathcal{L}_{\text{align}}$$ Typical training involves staged fine-tuning: starting from pure text-only SFT, introducing interleaved SFT, then optionally RL-based curriculum (“grouped chain reward” and outcome-based bonuses).

4. Dataset Construction and Domain Coverage

Large-scale benchmarks have been developed explicitly for interleaved CoT training and evaluation:

• Zebra-CoT (Li et al., 22 Jul 2025): 182,384 samples across domains (geometry, physics, games, maze navigation, etc.), alternating text and vision traces, curated via multi-stage pipelines combining synthetic and real samples.
• MINT-CoT (Chen et al., 5 Jun 2025): 54,031 mathematical items, each step aligned at the token level to a subset of image regions.
• ViTIB (Zhang et al., 14 Jul 2025): 1,382 video-QA items, each with 3+ key frames per video, human-verified visual-text coherence.
• ThinkMorph (Gu et al., 30 Oct 2025): 24,000 interleaved traces covering jigsaw assembly, chart refocus, navigation, and visual search.

Dataset curation typically involves: (1) automated/MLLM-based extraction of relevant visual content, (2) human or model-in-the-loop verification, (3) harmonization of text/image pairs, and (4) explicit alignment annotations to enable fine-grained supervision.

5. Empirical Benefits and Quantitative Performance

Interleaved CoT has demonstrated consistently superior performance to text-only or naïve multimodal approaches across a range of tasks:

• Zebra-CoT: +4.9 pp average accuracy gain (up to +13 pp on VisuLogic, +12.7 pp in-distribution geometry), strong OOD generalization (Li et al., 22 Jul 2025).
• ViTCoT: 8.6 pp average improvement over text-only CoT on 14-class video benchmarks, and 1.6 pp over direct (no-CoT) video reasoning (Zhang et al., 14 Jul 2025).
• MINT-CoT: +34.08% on MathVista, +28.78% on GeoQA, +23.2% on MMStar vs. corresponding baselines (Chen et al., 5 Jun 2025).
• OmniDrive-R1: +28.58 pp overall reasoning, +35.81 pp MCQ accuracy for vision-language autonomous driving (Zhang et al., 16 Dec 2025).
• ICoT: Up to +14% ROUGE-L gain over text-only CoT on visually-demanding QA (Gao et al., 29 Nov 2024).
• ThinkMorph: +34.7% in-domain average over base model, >20 pp OOD generalization gains, competitive with much larger VLMs (Gu et al., 30 Oct 2025).

Activation analyses reveal that interleaving increases the number of “active” attention heads (by 30–50 per sample, aggregated) and triggers deeper multimodal engagement compared to classical CoT (Zhang et al., 14 Jul 2025).

6. Interpretability, Emergent Properties, and Model Behavior

Interleaved CoT naturally enhances interpretability: explicit visual intermediates show “where” and “how” reasoning proceeds at each step, permitting human auditing and diagnosis. This contrasts with text-only rationales, which lack spatial grounding.

Models trained bi-directionally (both modalities) exhibit emergent behavior:

• Adaptive mode switching: selectively choosing purely textual, visual-only, or interleaved chains depending on domain and input (Gu et al., 30 Oct 2025).
• Unseen visual manipulations: spontaneous invention of new visual skills (e.g., zoom, overlay, multi-box) not explicitly present in training (Gu et al., 30 Oct 2025).
• Improved exploration/diversity: best-of-N sampling of interleaved trajectories enhances reasoning success rates, especially on perception-intensive tasks.

Plug-and-play mechanisms (e.g., ADS (Gao et al., 29 Nov 2024)) allow zero-latency interpretability boosts in existing VLMs without retraining, exposing attention-derived patch relevance at each reasoning juncture.

7. Limitations and Prospective Research Directions

Known limitations include:

• Supervision bottlenecks: Manual curation of trace quality and image-text alignment is expensive at scale (e.g., Zebra-CoT’s 182K traces (Li et al., 22 Jul 2025)).
• Domain adaptation challenges: Synthetic visual rationales may not transfer to real or specialized domains (e.g., textbook diagrams vs. generated sketches).
• Inference latency and model constraints: Vision–text decoding alternation requires architectural support and may increase compute time (Li et al., 22 Jul 2025, Gu et al., 30 Oct 2025).
• Failure modes: Over-application of visual steps, sensitivity to signal tokens, or ambiguous cross-modal correspondence can reduce performance (Gao et al., 29 Nov 2024, Gu et al., 30 Oct 2025).

Areas of ongoing research include:

• Dynamic interleaving policies: Learning to autonomously decide interleaving points and modalities rather than relying on fixed delimiters.
• Broader modality integration: Extending to video, audio, depth, and interactive tool use.
• Stronger alignment objectives: Employing contrastive/mutual-information losses for robust cross-modal fusion.
• Automated curriculum/data pipelines: Reducing manual data curation costs by developing automated metrics for trace and visual rationale coherence.
• Reinforcement learning with process-based rewards: Using proxy rewards (e.g., CLIP similarity) for annotation-free, real-time grounding (Zhang et al., 16 Dec 2025).

Interleaved Chain-of-Thought reasoning constitutes a foundational paradigm for advancing human-aligned, interpretable, and robust multimodal general intelligence (Zhang et al., 14 Jul 2025, Zou et al., 9 Oct 2025, Chen et al., 5 Jun 2025, Li et al., 22 Jul 2025, Zhang et al., 16 Dec 2025, Gao et al., 29 Nov 2024, Gu et al., 30 Oct 2025).