Cross-Modal Multi-Hop Reasoning (CMR)

• CMR is a computational approach that integrates heterogeneous data from multiple modalities using sequential multi-hop reasoning to synthesize complex information.
• It employs methods such as graph-based, transformer, and reinforcement learning strategies to iteratively connect and validate evidence across channels.
• Benchmarks demonstrate CMR’s potential in areas like multimodal QA and claim verification, though challenges in interpretability and reliability remain.

Searching arXiv for papers on cross-modal multi-hop reasoning and closely related benchmarks/frameworks. Cross-Modal Multi-Hop Reasoning (CMR) denotes the process of synthesizing and drawing inferences across different sensory modalities, such as text, images, audio, and video, through a sequence of intermediate reasoning steps rather than a single direct inference (Qian et al., 2024). In the question answering setting, it arises when information is scattered across multiple sources and modalities and answering a query requires gathering, relating, and aggregating heterogeneous evidence; in one concrete formulation, the task is: given a text question $Q$ and a set of candidate sources $\mathcal{S}_Q$ (text + images), select the subset $\mathcal{S}_Q^+$ that actually supports answering $Q$ (Yarrabelly et al., 7 Jan 2025). Across recent work, CMR has expanded from multimodal question answering over text, tables, and images (Talmor et al., 2021) to claim verification over text, images, and tables (Wang et al., 2024), financial reasoning over textual reports, tables, and charts (Kim et al., 2024), tri-modal reasoning over text, image, and speech (Kim et al., 22 Aug 2025), medical instructional video understanding with text, video, and structured knowledge (Liu et al., 6 Jul 2025), and long-form audio-visual reasoning with temporally dispersed evidence (Xu et al., 27 May 2026).

1. Conceptual scope and task formulations

CMR is distinguished by two coupled properties. The first is cross-modality: evidence is distributed across modalities and cannot be resolved by a single source alone. The second is multi-hop reasoning: the model must perform a sequence of inferential steps, each possibly requiring navigation across different modalities, to answer a query or complete a task (Qian et al., 2024). In survey form, this is expressed as

$$A = f(\phi_1(X_1), \phi_2(X_2), ..., \phi_n(X_n), Q)$$

with multi-hop settings modeled as a composition of reasoning steps,

$A = f_k(...f_2(f_1(X_1, Q), X_2)...)$

where each $f_j$ is a reasoning hop and may alter modality (Qian et al., 2024).

A related formalization appears in tri-modal evaluation. In "CMR-SPB" (Kim et al., 22 Aug 2025), CMR is modeled using modality-specific knowledge graphs in which entities have attributes accessible through text, image, or speech, and a reasoning path is denoted as an ordered sequence such as I-T or S-T-I. For $N$ modalities, the number of possible ordered paths is $N!$ (Kim et al., 22 Aug 2025). This formulation makes the sequencing of modality transitions explicit rather than treating multimodal fusion as a single undifferentiated operation.

Task formulations vary by domain. In MultiModalQA, complex questions require joint reasoning over text, tables and images, and the dataset was built through a formal language that composes single-modality questions into cross-modal questions (Talmor et al., 2021). In WebQA-style source retrieval, the target is not the final answer but the support set $\mathcal{S}_Q^+$ (Yarrabelly et al., 7 Jan 2025). In multi-hop multimodal claim verification, a system must aggregate and reason over all evidence across modalities to determine whether the evidence SUPPORTS or REFUTES a claim (Wang et al., 2024). In medical video understanding, M$\mathcal{S}_Q$0-Med defines Temporal Answer Grounding in Single Video (TAGSV) and Temporal Answer Grounding in Video Corpus (TAGVC), so the output is a temporal segment rather than a span, label, or entity (Liu et al., 6 Jul 2025). In image-to-point cloud registration, CMR-Agent reformulates the procedure as an iterative Markov decision process in which cross-modal reasoning is embodied in sequential pose refinement rather than symbolic answer generation (Yao et al., 2024).

This variety suggests that CMR is better understood as a reasoning regime than as a single benchmark family. A plausible implication is that progress in one subproblem—such as multimodal retrieval, graph traversal, or agentic planning—does not automatically transfer to all others unless the underlying bottleneck is shared.

2. Benchmarks, datasets, and evaluation settings

The benchmark literature has progressively tightened the requirement that multiple modalities and multiple hops are genuinely necessary. MultiModalQA introduced 29,918 questions over text, tables and images, and empirically demonstrated the necessity of a multi-modal multi-hop approach: ImplicitDecomp achieves an average F1 of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, while human performance is at 90.1 F1 (Talmor et al., 2021). Its construction uses a formal pseudo-language with operations such as Compose, Intersect, and Compare to combine single-modality questions into compositional multi-hop questions (Talmor et al., 2021).

WebQA, used in "Multimodal Multihop Source Retrieval for Web Question Answering" (Yarrabelly et al., 7 Jan 2025), is designed for multi-hop, multimodal QA, with each question paired with candidate text snippets and images labeled as positive or negative. The reported statistics are Train: 18,954 image questions, 3,464 text questions, with Dev/Test approximately 2,500–4,000 per modality, and F1 score as the evaluation metric due to severe class imbalance (Yarrabelly et al., 7 Jan 2025).

Recent benchmarks have focused increasingly on failure modes of existing evaluations. FCMR argues that existing benchmarks such as MMQA suffer from data contamination and a lack of complex queries that necessitate operations across more than two modalities; specifically, for MMQA, only 0.8% of queries need all three modalities and true 3-hop reasoning (Kim et al., 2024). FCMR therefore constructs a financial benchmark over textual reports, tables, and charts, divided into Easy, Medium, and Hard, with Hard requiring precise cross-modal three-hop reasoning and designed to prevent the disregard of any modality (Kim et al., 2024). The best-performing model, Claude 3.5 Sonnet, achieves only 30.4% accuracy on the most challenging tier (Kim et al., 2024).

MMCV introduces multi-hop multimodal claim verification and a large-scale dataset comprising 15k multi-hop claims paired with multimodal evidence from text, images, and tables (Wang et al., 2024). Claims range from single-hop to 4-hop, and the paper reports that the task is challenging even for the latest state-of-the-art multimodal LLMs, especially as the number of reasoning hops increases (Wang et al., 2024).

CMR-SPB extends the evaluation axis to speech and reasoning-path bias. It introduces a benchmark over text, image, and speech in which all six possible reasoning paths for three modalities are represented equally, approximately 400 examples each (Kim et al., 22 Aug 2025). The paper argues that existing benchmarks largely overlook the speech modality and exhibit heavily biased reasoning path distributions, which can severely undermine fair evaluation (Kim et al., 22 Aug 2025).

Domain-specific and temporally grounded settings further extend the benchmark landscape. M$\mathcal{S}_Q$1-Med is described as the first benchmark for Multi-lingual, Multi-modal, and Multi-hop reasoning in medical instructional video understanding, with 3,748 videos and 12,747 QA pairs, and questions explicitly labeled as simple or complex (Liu et al., 6 Jul 2025). MOV-Bench contains 519 carefully curated questions requiring multi-hop reasoning over temporally dispersed audio-visual evidence (Xu et al., 27 May 2026). CRIT introduces a graph-based automatic pipeline for generating complex cross-modal reasoning tasks across natural images, videos, and text-rich sources, with a manually verified test set for reliable evaluation (Sung et al., 2 Apr 2026).

Taken together, these benchmarks shift the evaluation target from multimodal perception alone to compositional evidence chaining, path balance, contamination resistance, temporal grounding, and domain specificity.

3. Methodological families

The methodological space of CMR is heterogeneous, but several recurring families can be identified from the literature.

Graph-based reasoning is central in settings where evidence relations can be encoded explicitly. In WebQA source retrieval, the task is modeled as node classification in a semantic graph, with nodes as question-conditioned modality sources and edges as semantic or contextual relationships (Yarrabelly et al., 7 Jan 2025). Graph topologies include Dense Super-Node, Star Graph, and gated variants (Yarrabelly et al., 7 Jan 2025). In knowledge graph reasoning, MMKGR combines a unified gate-attention network for multi-modal complementary features with a complementary feature-aware reinforcement learning method for multi-hop traversal (Zheng et al., 2022). RConE extends logical query answering on multi-modal knowledge graphs through rough convex cones and supports conjunction, disjunction, and negation over MMKGs (Kharbanda et al., 2024).

Programmatic, symbolic, and compositional reasoning appears wherever tasks admit explicit decomposition. MultiModalQA formalizes composition through a pseudo-language with operators such as Compose, Intersect, and Compare (Talmor et al., 2021). The survey on interpretable cross-modal reasoning identifies graph explanation and symbol explanation as two major explanatory forms, noting that multi-hop reasoning is often modeled as traversals in graphs or as modular program execution where each module corresponds to a reasoning step (Xue et al., 2023).

Transformer- and attention-based reasoning remains important, but recent work often critiques purely token-wise cross-attention as insufficient or inefficient for multi-source multi-hop settings. In WebQA, message propagation over graph networks or the entire graph structure can replace massive multimodal transformers with token-wise cross-attention (Yarrabelly et al., 7 Jan 2025). In COCO-MMR, Enigma-COT introduces multi-hop cross-modal attention: $\mathcal{S}_Q$2 with gated updates

$\mathcal{S}_Q$3

to iteratively refine text representations via image attention (Wei et al., 2023).

Agentic and retrieval-augmented reasoning has become prominent in recent multimodal LLM work. The survey on LLM-based CMR classifies methodologies into roles including Multimodal Fusion Engine, Textual Processor, Cognitive Controller, and Knowledge Enhancer (Qian et al., 2024). Cognitive-controller systems use programmatic construction, tool chaining, linguistic interaction, and external retrieval as reasoning hops (Qian et al., 2024). MMhops-R1 frames reasoning as a sequential decision process over actions such as think, image retrieval, text retrieval, and final answer generation, training a policy with a composite reward over outcome, format, and action usefulness (Zhang et al., 15 Dec 2025). AOP-Agent uses a hierarchical omni-modal memory plus an observe-reflect-replan loop for long-form audio-visual reasoning (Xu et al., 27 May 2026).

Reasoning-trace distillation and prompt-structured inference occupy an intermediate position between pure prompting and full architectural redesign. SDRT uses a dual-query reasoning procedure to derive reasoning-guided responses, followed by self-distillation with an intervention adapter, a cross-modal skip connection, and an ensemble of diverse reasoning traces (Wu et al., 3 Mar 2025). CMR-SPB introduces ECV prompting—Extract, Connect, Verify—motivated by the observation that entity linking is the primary bottleneck (Kim et al., 22 Aug 2025).

These families are not mutually exclusive. Recent systems increasingly combine graph or memory structures, retrieval and tool use, and structured prompting or policy learning.

4. Multi-hop mechanisms and cross-modal evidence integration

At the algorithmic level, CMR systems differ most sharply in how they represent state, propagate information, and control reasoning depth.

In graph neural approaches, multi-hop reasoning is implemented through message passing. WebQA source retrieval uses standard aggregation,

$\mathcal{S}_Q$4

or gated propagation,

$\mathcal{S}_Q$5

where the gate $\mathcal{S}_Q$6 is learned as a function of both node embeddings (Yarrabelly et al., 7 Jan 2025). The key claim of that work is that both graph structure and adjacency matrix are task-related prior knowledge, and graph structure can be leveraged to improve retrieval performance (Yarrabelly et al., 7 Jan 2025).

In reinforcement-learning formulations, multi-hop reasoning is cast as sequential decision making. MMKGR defines states around the current entity, query, and neighborhood, and rewards the agent with a 3D reward composed of destination reward, distance reward, and diverse reward to address sparse reward and encourage short, diverse, successful paths (Zheng et al., 2022). CMR-Agent similarly uses an iterative MDP for 2D–3D registration, with state $\mathcal{S}_Q$7, action $\mathcal{S}_Q$8, and reward based on improvement in point-to-point alignment (Yao et al., 2024). Its hybrid state representation combines 2D projected features and 3D frustum-alignment cues: $\mathcal{S}_Q$9 to address neutral states caused by the spatial truncation of camera frustum (Yao et al., 2024).

In retrieval-augmented multimodal LLM systems, the hop mechanism is explicit in tool invocation. MMhops-R1 defines a trajectory

$\mathcal{S}_Q^+$0

and a reward

$\mathcal{S}_Q^+$1

so that planning, retrieval, and synthesis are jointly optimized (Zhang et al., 15 Dec 2025). AOP-Agent uses a planner, observation tools, reflector, and reasoner, supported by working and evidence memories, to perform active perception over sparse and temporally dispersed audio-visual evidence (Xu et al., 27 May 2026).

Path-based evaluation has clarified that extraction alone is not the main challenge. CMR-SPB reports that on tasks solvable using just one modality, or all given but only one relevant, models perform near-perfectly (>90% accuracy); the struggle is in integrative multi-hop reasoning (Kim et al., 22 Aug 2025). Its step-by-step analysis shows that most failures occur in connecting two entities rather than in end-stage attribute extraction (Kim et al., 22 Aug 2025). FCMR reaches a similar conclusion in a different domain: the critical bottleneck is the information retrieval phase, with models often knowing which modality to use but failing to correctly extract precise details from charts or complex tables (Kim et al., 2024). CRIT’s manual error analysis identifies evidence localization error as the most frequent category, at 55% of audited GPT-4o mistakes (Sung et al., 2 Apr 2026).

This convergence across benchmarks suggests that the main systems problem in CMR is often not modality encoding in isolation, but faithful localization and chaining of complementary evidence under path constraints.

5. Empirical findings and comparative results

Reported results consistently show that CMR remains far from solved, especially as hop count, path complexity, or modality diversity increases.

On WebQA source retrieval, the VLP-VinVL transformer baseline achieves 68.1 on image, 69.5 on text, and 68.9 combined F1, whereas graph-based methods achieve lower absolute scores but remain competitive: Dense SuperNode 62.8 combined, Star Graph 63.1, SuperNode Gated Graph 54.5, and Star Gated Graph 63.4 (Yarrabelly et al., 7 Jan 2025). The paper also states that the proposed method shows a performance gain of 4.6% retrieval F1score over the transformer baselines, despite being a very light model (Yarrabelly et al., 7 Jan 2025). Since the details section reports the main comparison table with graph models below VLP-VinVL on the combined split, this apparent discrepancy should be read cautiously; a plausible interpretation is that the 4.6% gain refers to a different retrieval setting or baseline configuration discussed elsewhere in the paper (Yarrabelly et al., 7 Jan 2025).

MultiModalQA demonstrates the value of explicit multi-hop modeling. On the test set, AutoRouting obtains 38.2 F1 on multi-modal questions, while ImplicitDecomp reaches 51.7; human performance is 90.1 (Talmor et al., 2021). The question type classifier achieves 91.5% accuracy, and for compositional questions, correct first-hop predictions substantially improve second-hop performance (Talmor et al., 2021).

In knowledge graph reasoning, MMKGR reports strong gains over prior approaches on WN9-IMG-TXT and FB-IMG-TXT, with MRR 80.2 and 71.3 respectively, compared with 62.4 and 50.6 for RLH (Zheng et al., 2022). RConE reports dramatic gains on sub-entity queries and slight trade-offs on classic entity queries while remaining on par or better than BetaE and only moderately lower than ConE (Kharbanda et al., 2024).

Recent multimodal LLM evaluations are markedly more pessimistic. In MMCV open-book claim verification, GPT-4o, Gemini, and LLaVA show declining F1 as hop count increases; for example, Gemini achieves 79.20 on 1-hop, 71.66 on 2-hop, 65.86 on 3-hop, and 66.97 on 4-hop, while human annotators remain far ahead, around 83–85% on 4-hop (Wang et al., 2024). FCMR reports that proprietary MLLMs are markedly higher on Easy but only 30.4% on Hard for Claude 3.5 Sonnet, with random selection at approximately 12.3% (Kim et al., 2024). In M$\mathcal{S}_Q^+$2-Med, humans reach approximately 0.80 mIoU, while the best models remain below 0.50 for complex questions (Liu et al., 6 Jul 2025). In MOV-Bench, direct inference with Qwen3-Omni-Instruct scores 52.8%, while AOP-Agent raises this to 62.6%, and on long videos from 45.5% to 60.9% (Xu et al., 27 May 2026).

Training on targeted CMR data has shown substantial transfer benefits. CRIT reports that Qwen2.5-VL-7B improves from F1 29.1 to 59.5 on natural images, from 27.8 to 42.2 on video frames, and from 9.6 to 22.5 on scientific papers after CRIT training (Sung et al., 2 Apr 2026). Adding CRIT to the training mix raises SPIQA METEOR from 3.60 to 10.53, MMQA EM from 27.3 to 30.0, VEGA ROUGE-L from 29.5 to 35.1, and FCMR F1 from 44.9 to 50.5 (Sung et al., 2 Apr 2026).

These findings support a consistent picture: narrow tasks with explicit structure can be improved substantially by graph or multi-hop modeling, but robust general-purpose CMR across realistic modalities remains challenging even for strong multimodal LLMs.

6. Interpretability, reliability, and open problems

Interpretability is a recurring concern because multi-hop chains amplify the cost of hidden failure. The survey on interpretable cross-modal reasoning organizes explanation methods into visual, textual, graph, symbol, and multimodal explanation, emphasizing that multi-hop interpretability enables tracing outputs to specific steps and evidence chains (Xue et al., 2023). Graph and program approaches are especially amenable to explicit hop-level explanations, while chain-of-thought-style rationales make reasoning sequences legible to end users (Xue et al., 2023).

Reliability problems appear in several forms. The LLM survey identifies hallucination and reliability, interpretability, modality scalability, and computational cost as core limitations of LLM-based CMR systems (Qian et al., 2024). MMCV reports overconfidence, visual misinterpretation, temporal and logical reasoning failures, and hallucination risks in both modeling and dataset construction (Wang et al., 2024). FCMR shows that models often default to false when uncertain, struggle particularly with charts and with negative-number comparisons, and may skip required modalities (Kim et al., 2024). M$\mathcal{S}_Q^+$3-Med notes malformed output and refusal answers in general-purpose multimodal LLMs (Liu et al., 6 Jul 2025).

Benchmark design itself is now treated as a research problem. CMR-SPB argues that biased benchmarks risk misrepresenting model performance and proposes the Path Balance Score

$\mathcal{S}_Q^+$4

to combine mean accuracy with variance across reasoning paths (Kim et al., 22 Aug 2025). CRIT argues that most existing multimodal benchmarks fail to capture true multi-hop cross-modal ability because questions are often solvable using a single modality alone (Sung et al., 2 Apr 2026). FCMR similarly emphasizes contamination resistance and strict enforcement of three-hop cross-modal reasoning (Kim et al., 2024).

Several forward directions recur across papers. These include richer graph structures and edge types (Yarrabelly et al., 7 Jan 2025), better encoders and cross-modal representations (Yarrabelly et al., 7 Jan 2025), improved chart/table/text grounding (Kim et al., 2024), robust cross-modal linkage skills rather than only extraction (Kim et al., 22 Aug 2025), comprehensive and path-balanced benchmarks as a new gold standard (Kim et al., 22 Aug 2025), and training methods that instill dynamic planning and retrieval policies (Zhang et al., 15 Dec 2025). Agentic active-perception work further suggests that iterative observe–reflect–replan loops may be necessary when evidence is sparse, temporally dispersed, and distributed across modalities (Xu et al., 27 May 2026).

A broad synthesis of the literature is that CMR should not be reduced to multimodal fusion. The accumulated evidence indicates that its central technical difficulty lies in enforcing faithful intermediate structure: locating evidence, connecting entities across modalities, controlling hop sequences, and preserving grounding as reasoning depth increases. This suggests that future progress will depend as much on structured evaluation and controllable reasoning procedures as on larger foundation models alone.