X-CoT: Cross-Modal Chain-of-Thought

• X-CoT is a multimodal reasoning framework that integrates intermediate steps from different modalities such as text, vision, and audio to improve decision-making.
• It employs both explicit human-readable rationales and latent continuous thought states, bridging modality-specific evidence with fused inference.
• X-CoT underpins applications in medical imaging, video retrieval, and audio analysis by providing transparent, efficient, and verifiable reasoning pipelines.

Cross-Modal Chain-of-Thought (X-CoT) denotes explicit or latent intermediate reasoning that spans more than one modality—typically text plus vision, video, audio, tables, speech, or 3D data—in order to connect perception, grounding, fusion, and final inference within a structured chain. In the broader Chain-of-X formulation, it extends the canonical Chain-of-Thought pattern from textual intermediate steps to pipelines of multimodal intermediates, augmentations, feedback loops, or collaborating models, often represented as $$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$ (Xia et al., 2024). In later survey terminology, X-CoT is treated as equivalent to Multimodal Chain-of-Thought (MCoT), with explicit probabilistic factorization over intra-modal steps and fusion steps to model how modality-specific evidence evolves and is integrated over time (Zhu et al., 17 Nov 2025). Across the recent literature, X-CoT has developed along three main lines: human-readable textual rationales, interleaved visual-textual or modal-mixed traces, and latent-space reasoning that replaces natural-language rationales with continuous thought states (Pham et al., 18 Aug 2025).

1. Conceptual definition and taxonomy

X-CoT is motivated by the observation that multimodal tasks often require both perception and reasoning, and that a purely textual scratchpad is frequently insufficient for maintaining alignment to visual, auditory, tabular, or temporal evidence. The survey literature characterizes its purpose as breaking multimodal perception into manageable intermediate representations, grounding reasoning in modality-specific evidence such as regions, video segments, table operations, or phonetic units, and creating explicit intervention points for retrieval, verification, feedback, or tool use (Xia et al., 2024).

Within Chain-of-X taxonomies, X-CoT is not treated as a single algorithmic template but as a family of designs spanning several node types. The survey places it primarily under Multi-Modal Interaction and LLMs as Agents, and describes implementations through Chain-of-Intermediates, Chain-of-Augmentation, Chain-of-Feedback, and Chain-of-Models (Xia et al., 2024). A later survey reformulates the same area through multimodal rationale generation, program-of-thought and tool calls, decomposition and planning, visual grounding-aware CoT, multi-hop cross-modal reasoning, and self-reflection or verification mechanisms (Zhu et al., 17 Nov 2025).

A key distinction in this literature is between explicit and implicit reasoning. Explicit X-CoT emits readable rationales, supports process supervision, and exposes failure points for diagnosis and auditing; latent variants keep intermediate states internal, trading off interpretability for compactness and inference efficiency (Zhu et al., 17 Nov 2025). This distinction underlies many later methodological divergences, including continuous-thought models, modal-mixed latent embeddings, and inference-time decoding schemes that attempt to preserve grounding without generating long textual chains (Pham et al., 18 Aug 2025).

2. Core mechanisms and formal structure

The survey literature provides two complementary formalizations. At the general Chain-of-X level, the chain is written as $$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$, with the $X_i$ allowed to be intermediates, retrieved evidence, feedback, or model outputs rather than only textual thoughts (Xia et al., 2024). In the multimodal survey, text-only CoT is extended to modalities $m \in M$ through intra-modal steps $r_t^m$ and fused steps $r_t$, yielding a decomposition in which modality-specific states evolve over time and are then fused into a shared reasoning state before final prediction (Zhu et al., 17 Nov 2025).

Many concrete X-CoT systems instantiate this general idea through a perception stage, an intermediate grounding stage, and a reasoning stage. In medical imaging, X-Ray-CoT extracts visual features with a ViT encoder, identifies radiologically meaningful concepts, aligns image and concept text through a visual-language alignment layer, and then uses an LLM with a structured prompt to reason through abnormalities, pathophysiology, diagnosis, and justification (Ng et al., 17 Aug 2025). In 3D vision-language learning, CoT enters on the textual side of contrastive alignment as hierarchical supervision over object recognition, functional inference, and causal reasoning, thereby making the shared embedding sensitive to the reasoning process rather than only to static captions (Chen et al., 8 Mar 2025).

The same pattern appears in retrieval-oriented systems, but with ranking replacing generation as the terminal task. In X-CoT for text-to-video retrieval, the baseline cosine score

$$s(q, v) = \frac{\langle \mathbf{z}_q, \mathbf{z}_v \rangle}{\|\mathbf{z}_q\|\,\|\mathbf{z}_v\|}$$

is replaced by pairwise LLM judgments over structured video annotations, and the resulting local preferences are aggregated globally through a Bradley–Terry model

$$P(v_i \succ v_j) = \frac{\theta_i}{\theta_i + \theta_j}$$

to obtain a complete ranking (Pulakurthi et al., 25 Sep 2025). In CIR-CoT for composed image retrieval, the model first generates a three-stage reasoning trace—Caption, Reasoning, Conclusion—and then encodes the final retrieval intent into a special <emb> token whose hidden state is used for retrieval under an InfoNCE objective (Lin et al., 9 Oct 2025).

3. Major implementation paradigms

A first paradigm is explicit natural-language reasoning. This remains the dominant form in many systems because it supports direct process supervision and human-readable explanations. X-Ray-CoT explicitly guides the model through “Identify Abnormalities,” “Infer Pathophysiology,” “Synthesize and Diagnose,” and “Justify Diagnosis,” producing detailed reports that reference visual concepts and medical knowledge (Ng et al., 17 Aug 2025). Audio-CoT applies Manual-CoT, Zero-Shot-CoT, Description-first CoT, and Self-Consistency entirely at inference time to audio-LLMs, showing that the CoT prompt can be transferred from text to sound, music, and speech reasoning without retraining (Ma et al., 13 Jan 2025). CIR-CoT likewise requires an end-to-end retrieval model to emit an interpretable reasoning chain before producing its retrieval embedding (Lin et al., 9 Oct 2025).

A second paradigm is interleaved multimodal reasoning, where textual steps are paired with visual or other non-textual evidence inside the chain itself. VTI-CoT defines an interleaved sequence $R=\{(s_k,v_k)\}_{k=1}^K$ for video reasoning, where each textual step is grounded to a temporally bounded frame or clip interval. To avoid extremely long multimodal token sequences, it renders the entire visual-textual chain into a single OCR-style canvas and supervises designated visual-token slots through hidden-state alignment to frozen vision-encoder features (Zhang et al., 4 Jun 2026). Modal-mixed CoT extends the same principle to latent visual sketches delimited by control tokens such as $\langle START\rangle$ and $$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$0, so that a reasoning sequence can interleave text and compact visual latent blocks generated by a diffusion-based decoder (Shao et al., 31 Jan 2026).

A third paradigm is latent-space reasoning. MCOUT replaces discrete textual rationales with a “continuous thought” vector that is iteratively appended to the sequence and refined, either by reusing the LM’s last hidden state or by applying multimodal latent attention over image embeddings (Pham et al., 18 Aug 2025). CoLT similarly teaches a multimodal model to reason through a small number of latent thought vectors, using an external decoder for forward and backward supervision during training and removing that decoder at inference time; the reported effect is to reduce inference time relative to text CoT while maintaining a semantically structured latent chain (Hu et al., 30 Jun 2026). L2V-CoT goes further by treating CoT as a transferable low-frequency latent direction: it extracts contrastive CoT representations from an LLM with Linear Artificial Tomography, rescales them in the frequency domain, and injects them into a VLM at inference time without additional training (Zhan et al., 22 Nov 2025).

4. Retrieval-centered X-CoT

Retrieval is one of the clearest settings in which X-CoT is used not only to improve performance but also to convert black-box similarity scores into inspectable decisions. In text-to-video retrieval, conventional systems encode queries and videos into a joint embedding space and rank candidates by cosine similarity, but this leaves no explanation for why a match is good or poor and makes low-quality text–video pairs difficult to identify (Pulakurthi et al., 25 Sep 2025). X-CoT addresses this by constructing structured annotations from sampled frames—object tags, action tags, scene tags, and a temporal summary—and prompting an LLM to perform pairwise comparisons between candidate videos with stepwise reasoning over objects, actions, scenes, temporal relations, and mismatches. Pairwise outputs are then aggregated with a Bradley–Terry model after sliding-window refinement over the top-$$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$1 list (Pulakurthi et al., 25 Sep 2025).

The empirical results reported for this system show consistent reranking gains across MSR-VTT, MSVD, DiDeMo, and LSMDC. On MSR-VTT, for example, R@1 improves from 31.6 to 33.7 for CLIP, from 36.4 to 37.2 for VLM2Vec, and from 46.9 to 47.3 for X-Pool; on MSVD, the corresponding gains are 36.5 to 42.1, 46.7 to 48.4, and 47.2 to 49.1 (Pulakurthi et al., 25 Sep 2025). The same study also reports that perturbing 20% of tags causes only a small drop, illustrated by MSR-VTT R@1 changing from 33.7 to 32.3, and that explanation–decision alignment rises from 0.273 to 0.350, suggesting greater semantic faithfulness between rationales and final selections (Pulakurthi et al., 25 Sep 2025).

A parallel development appears in composed image retrieval. CIR-CoT treats the retrieval input as a reference image plus a modification instruction, supervises a structured CoT trace through Caption, Reasoning, and Conclusion, and then extracts a retrieval-intent embedding from a special token (Lin et al., 9 Oct 2025). On CIRR test data, the Full variant reports R@1 55.06, R@5 85.47, R@10 92.60, and Avg. 82.49, while on FashionIQ validation it reports average R@10 56.29 and R@50 76.42; in zero-shot transfer from CIRR supervision to CIRCO, it reports mAP@5 33.54 and mAP@50 37.29 (Lin et al., 9 Oct 2025). Together, these retrieval systems establish a recurring X-CoT pattern: embeddings remain useful for candidate generation or final matching, but the crucial ranking decision is mediated by an interpretable reasoning trace rather than an opaque scalar alone.

5. Grounding, faithfulness, and interpretability

A central claim across X-CoT research is that intermediate reasoning should remain tied to modality-specific evidence rather than drifting into purely verbal elaboration. Several systems implement this through structured inputs. X-CoT for text-to-video retrieval avoids direct use of benchmark ground-truth captions and instead feeds curated annotations derived from frames, with deduplication, normalization, and aggregation designed to reduce spurious lexical signals (Pulakurthi et al., 25 Sep 2025). X-Ray-CoT grounds its report generation in detected visual concepts such as opacities or cardiomegaly before invoking medical reasoning, and its final reports explicitly include “Detailed Diagnostic Reasoning and Basis” and “Relevant Visual Concepts Observed” (Ng et al., 17 Aug 2025). VTI-CoT forces each reasoning step to cite temporal interval tags and pairs it with deterministically retrieved visual evidence during data construction (Zhang et al., 4 Jun 2026).

Even when rationales are generated, however, later work shows that models may not actually use them. RED identifies a specific failure mode in large vision-LLMs: standard multimodal CoT often generates a rationale $$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$2 and then decodes the final answer while effectively ignoring the rationale’s content because image tokens dominate the answer distribution (Yamaguchi et al., 10 Jul 2025). RED reformulates answer decoding as KL-constrained reward maximization, where the image-conditional distribution acts as a reference policy and the rationale-conditional log-likelihood acts as a reward. This yields a product-of-experts next-token distribution

$$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$3

which the paper argues enforces token-level dependence on both visual evidence and rationale semantics (Yamaguchi et al., 10 Jul 2025). Intervention experiments reported there show that better rationales help more under RED, while randomly swapped rationales degrade RED substantially, indicating restored faithfulness to rationale content (Yamaguchi et al., 10 Jul 2025).

The field’s interpretability claims are therefore increasingly differentiated. Some systems emphasize readable reports or rationales; others emphasize explanation–decision alignment or rationale-conditioned decoding; latent models instead propose indirect diagnostics such as cross-modal attention maps, per-step latent statistics, or offline decoding of latent states back into text (Pham et al., 18 Aug 2025). This suggests that “interpretability” in X-CoT is not a single property but a spectrum ranging from natural-language traceability to analyzable internal trajectories.

6. Empirical benefits, limitations, and controversies

The empirical record for X-CoT is mixed rather than uniformly positive. On the benefit side, several task-specific systems report gains over non-CoT baselines. X-Ray-CoT reports Balanced Accuracy 80.52% and F1 78.65% for disease diagnosis on CORDA, slightly above the best black-box baseline at 80.00% Balanced Accuracy, while also generating detailed reports (Ng et al., 17 Aug 2025). MCOUT reports up to 8.23% accuracy gains and up to 8.27 BLEU improvements over its baseline across ScienceQA, MMMU, and MMStar (Pham et al., 18 Aug 2025). CoLT reports average accuracy 79.1% versus 75.7% for text CoT and 69.5% for direct answering on eight benchmarks with a Qwen3-VL-8B backbone, alongside 10.1$$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$4 end-to-end and 22.6$$\langle \text{input}, X_1, \dots, X_n, \text{output}\rangle$$5 decoding speedups relative to text CoT (Hu et al., 30 Jun 2026). L2V-CoT reports average improvement of 3.7% and up to 8.6% across multimodal reasoning benchmarks while remaining training-free (Zhan et al., 22 Nov 2025).

At the same time, systematic evaluation shows that multimodal CoT is not beneficial in all regimes. The study “Look Light, Think Heavy” reports that CoT tends to hurt on perception-heavy tasks, with average drops of 4.6% on visual grounding, 3.3% on knowledge-based VQA, and 4.8% on object counting, while helping mathematical reasoning by 6.1%, scientific reasoning by 2.9%, and multi-image reasoning by 4.9% (Jin et al., 21 Jun 2026). The same study identifies a “Look Light, Think Heavy” pattern in which verbal reflection rises and then falls over the reasoning trace, while visual reflection steadily declines, and probe-based analyses show that average accuracy on visual reasoning probes is 20% lower than on textual reasoning probes (Jin et al., 21 Jun 2026). This finding directly challenges the common assumption that more verbal reasoning necessarily implies better multimodal grounding.

Other limitations recur across subfields. X-CoT for text-to-video retrieval notes dependence on LLM capability, residual reliance on noisy annotations, temporal reasoning challenges, and lack of explicit evaluation for multilingual or adversarial inputs (Pulakurthi et al., 25 Sep 2025). Audio-CoT finds that CoT helps easy and medium tasks but degrades hard tasks, where long or speculative chains can confuse the model rather than improve accuracy (Ma et al., 13 Jan 2025). MCOUT observes modality collapse in multimodal latent attention, with weak visual contribution and a need for normalization to avoid norm imbalance (Pham et al., 18 Aug 2025). Modal-mixed latent reasoning notes that fixed latent block length may be suboptimal and that RL-based modality switching remains imperfect (Shao et al., 31 Jan 2026). A plausible implication is that X-CoT is best understood as a family of controllable reasoning interfaces whose value depends strongly on task type, supervision quality, and whether the system can preserve continual access to visual or other non-textual evidence.

7. Research directions and broader significance

Recent work points toward three major future directions. The first is better grounding and verification. Survey papers repeatedly call for causal analysis of intermediate nodes, stronger process supervision, region- or segment-level binding, verification chains, and tool-augmented reasoning with OCR, detectors, retrieval, or specialized analyzers (Xia et al., 2024). The limitations study recommends explicit visual check-backs, region-level evidence per step, and stronger support for crop, zoom, and other visual tools so that long CoTs do not become increasingly text-dominated (Jin et al., 21 Jun 2026).

The second direction is efficiency through latent or compressed reasoning. MCOUT, modal-mixed CoT, CoLT, and L2V-CoT all respond to the token inefficiency of textual rationales by shifting reasoning into latent vectors, visual sketches, or transferred latent directions (Pham et al., 18 Aug 2025). VTI-CoT addresses the same issue at the supervision level through OCR-based canvas compression, replacing highly variable long multimodal token sequences with a fixed visual-token budget (Zhang et al., 4 Jun 2026). This suggests a broad movement from explicit narrative CoT toward compact, model-native intermediates that retain some of CoT’s decompositional benefits while reducing latency and memory costs.

The third direction is application-specific specialization. Retrieval systems use X-CoT to expose ranking rationales and dataset flaws (Pulakurthi et al., 25 Sep 2025); medical systems use it to make diagnostic logic clinically inspectable (Ng et al., 17 Aug 2025); audio systems use it to structure perceptual cues before reasoning (Ma et al., 13 Jan 2025); 3D vision-language systems use it to bind geometric structure to affordances and causal interactions (Chen et al., 8 Mar 2025). Across these domains, the common pattern is not merely “reason step by step,” but rather “reason through intermediates that preserve contact with the modality-specific evidence required by the task.” This suggests that the enduring significance of X-CoT lies less in any single prompting formula than in its role as a general design principle for turning multimodal systems from end-to-end black boxes into inspectable, modifiable pipelines whose intermediate states can be grounded, evaluated, and, increasingly, compressed or transferred across architectures.