---
title: Cross-Modal Multi-Hop Reasoning (CMR)
url: https://www.emergentmind.com/topics/cross-modal-multi-hop-reasoning-cmr
type: topic
---

# Cross-Modal Multi-Hop Reasoning (CMR)

Searching arXiv for recent papers on cross-modal multi-hop reasoning and closely related benchmarks/frameworks.
Cross-Modal Multi-Hop Reasoning (CMR) denotes the process of synthesizing and drawing inferences across different sensory modalities, such as text, images, audio, and video, through a sequence of intermediate reasoning steps rather than a single direct inference [2409.18996]. In the question answering setting, it arises when information is scattered across multiple sources and modalities and answering a query requires gathering, relating, and aggregating heterogeneous evidence; in one concrete formulation, the task is: given a text question $Q$ and a set of candidate sources $\mathcal{S}_Q$ (text + images), select the subset $\mathcal{S}_Q^+$ that actually supports answering $Q$ [2501.04173]. Across recent work, CMR has expanded from multimodal question answering over text, tables, and images [2104.06039] to claim verification over text, images, and tables [2411.09547], financial reasoning over textual reports, tables, and charts [2412.12567], tri-modal reasoning over text, image, and speech [2508.16198], medical instructional video understanding with text, video, and structured knowledge [2507.04289], and long-form audio-visual reasoning with temporally dispersed evidence [2605.28192].

## 1. Conceptual scope and task formulations

CMR is distinguished by two coupled properties. The first is **cross-modality**: evidence is distributed across modalities and cannot be resolved by a single source alone. The second is **multi-hop reasoning**: the model must perform a sequence of inferential steps, each possibly requiring navigation across different modalities, to answer a query or complete a task [2409.18996]. In survey form, this is expressed as
\[
A = f(\phi_1(X_1), \phi_2(X_2), ..., \phi_n(X_n), Q)
\]
with multi-hop settings modeled as a composition of reasoning steps,
\[
A = f_k(...f_2(f_1(X_1, Q), X_2)...)
\]
where each $f_j$ is a reasoning hop and may alter modality [2409.18996].

A related formalization appears in tri-modal evaluation. In "CMR-SPB" [2508.16198], CMR is modeled using modality-specific knowledge graphs in which entities have attributes accessible through text, image, or speech, and a reasoning path is denoted as an ordered sequence such as I-T or S-T-I. For $N$ modalities, the number of possible ordered paths is $N!$ [2508.16198]. This formulation makes the sequencing of modality transitions explicit rather than treating multimodal fusion as a single undifferentiated operation.

Task formulations vary by domain. In MultiModalQA, complex questions require joint reasoning over text, tables and images, and the dataset was built through a formal language that composes single-modality questions into cross-modal questions [2104.06039]. In WebQA-style source retrieval, the target is not the final answer but the support set $\mathcal{S}_Q^+$ [2501.04173]. In multi-hop multimodal claim verification, a system must aggregate and reason over all evidence across modalities to determine whether the evidence SUPPORTS or REFUTES a claim [2411.09547]. In medical video understanding, M$^3$-Med defines Temporal Answer Grounding in Single Video (TAGSV) and Temporal Answer Grounding in Video Corpus (TAGVC), so the output is a temporal segment rather than a span, label, or entity [2507.04289]. In image-to-point cloud registration, CMR-Agent reformulates the procedure as an iterative Markov decision process in which cross-modal reasoning is embodied in sequential pose refinement rather than symbolic answer generation [2408.02394].

This variety suggests that CMR is better understood as a reasoning regime than as a single benchmark family. A plausible implication is that progress in one subproblem—such as multimodal retrieval, graph traversal, or agentic planning—does not automatically transfer to all others unless the underlying bottleneck is shared.

## 2. Benchmarks, datasets, and evaluation settings

The benchmark literature has progressively tightened the requirement that multiple modalities and multiple hops are genuinely necessary. MultiModalQA introduced 29,918 questions over text, tables and images, and empirically demonstrated the necessity of a multi-modal multi-hop approach: ImplicitDecomp achieves an average F1 of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, while human performance is at 90.1 F1 [2104.06039]. Its construction uses a formal pseudo-language with operations such as `Compose`, `Intersect`, and `Compare` to combine single-modality questions into compositional multi-hop questions [2104.06039].

WebQA, used in "Multimodal Multihop Source Retrieval for Web Question Answering" [2501.04173], is designed for multi-hop, multimodal QA, with each question paired with candidate text snippets and images labeled as positive or negative. The reported statistics are Train: 18,954 image questions, 3,464 text questions, with Dev/Test approximately 2,500–4,000 per modality, and F1 score as the evaluation metric due to severe class imbalance [2501.04173].

Recent benchmarks have focused increasingly on failure modes of existing evaluations. FCMR argues that existing benchmarks such as MMQA suffer from data contamination and a lack of complex queries that necessitate operations across more than two modalities; specifically, for MMQA, only 0.8% of queries need all three modalities and true 3-hop reasoning [2412.12567]. FCMR therefore constructs a financial benchmark over textual reports, tables, and charts, divided into Easy, Medium, and Hard, with Hard requiring precise cross-modal three-hop reasoning and designed to prevent the disregard of any modality [2412.12567]. The best-performing model, Claude 3.5 Sonnet, achieves only 30.4% accuracy on the most challenging tier [2412.12567].

MMCV introduces multi-hop multimodal claim verification and a large-scale dataset comprising 15k multi-hop claims paired with multimodal evidence from text, images, and tables [2411.09547]. Claims range from single-hop to 4-hop, and the paper reports that the task is challenging even for the latest state-of-the-art multimodal large language models, especially as the number of reasoning hops increases [2411.09547].

CMR-SPB extends the evaluation axis to speech and reasoning-path bias. It introduces a benchmark over text, image, and speech in which all six possible reasoning paths for three modalities are represented equally, approximately 400 examples each [2508.16198]. The paper argues that existing benchmarks largely overlook the speech modality and exhibit heavily biased reasoning path distributions, which can severely undermine fair evaluation [2508.16198].

Domain-specific and temporally grounded settings further extend the benchmark landscape. M$^3$-Med is described as the first benchmark for Multi-lingual, Multi-modal, and Multi-hop reasoning in medical instructional video understanding, with 3,748 videos and 12,747 QA pairs, and questions explicitly labeled as simple or complex [2507.04289]. MOV-Bench contains 519 carefully curated questions requiring multi-hop reasoning over temporally dispersed audio-visual evidence [2605.28192]. CRIT introduces a graph-based automatic pipeline for generating complex cross-modal reasoning tasks across natural images, videos, and text-rich sources, with a manually verified test set for reliable evaluation [2604.01634].

Taken together, these benchmarks shift the evaluation target from multimodal perception alone to compositional evidence chaining, path balance, contamination resistance, temporal grounding, and domain specificity.

## 3. Methodological families

The methodological space of CMR is heterogeneous, but several recurring families can be identified from the literature.

**Graph-based reasoning** is central in settings where evidence relations can be encoded explicitly. In WebQA source retrieval, the task is modeled as node classification in a semantic graph, with nodes as question-conditioned modality sources and edges as semantic or contextual relationships [2501.04173]. Graph topologies include Dense Super-Node, Star Graph, and gated variants [2501.04173]. In knowledge graph reasoning, MMKGR combines a unified gate-attention network for multi-modal complementary features with a complementary feature-aware reinforcement learning method for multi-hop traversal [2209.01416]. RConE extends logical query answering on multi-modal knowledge graphs through rough convex cones and supports conjunction, disjunction, and negation over MMKGs [2408.11526].

**Programmatic, symbolic, and compositional reasoning** appears wherever tasks admit explicit decomposition. MultiModalQA formalizes composition through a pseudo-language with operators such as `Compose`, `Intersect`, and `Compare` [2104.06039]. The survey on interpretable cross-modal reasoning identifies graph explanation and symbol explanation as two major explanatory forms, noting that multi-hop reasoning is often modeled as traversals in graphs or as modular program execution where each module corresponds to a reasoning step [2309.01955].

**Transformer- and attention-based reasoning** remains important, but recent work often critiques purely token-wise cross-attention as insufficient or inefficient for multi-source multi-hop settings. In WebQA, message propagation over graph networks or the entire graph structure can replace massive multimodal transformers with token-wise cross-attention [2501.04173]. In COCO-MMR, Enigma-COT introduces multi-hop cross-modal attention:
\[
A^k = \mathrm{MHA}^k(H_t^{k-1}, H_i, H_i)
\]
with gated updates
\[
H_t^k = (1 - G^k) \odot H_t^{k-1} + G^k \odot A^k,
\quad
G^k = \sigma(W^k[H_t^{k-1}, A^k])
\]
to iteratively refine text representations via image attention [2307.12626].

**Agentic and retrieval-augmented reasoning** has become prominent in recent multimodal LLM work. The survey on LLM-based CMR classifies methodologies into roles including Multimodal Fusion Engine, Textual Processor, Cognitive Controller, and Knowledge Enhancer [2409.18996]. Cognitive-controller systems use programmatic construction, tool chaining, linguistic interaction, and external retrieval as reasoning hops [2409.18996]. MMhops-R1 frames reasoning as a sequential decision process over actions such as think, image retrieval, text retrieval, and final answer generation, training a policy with a composite reward over outcome, format, and action usefulness [2512.13573]. AOP-Agent uses a hierarchical omni-modal memory plus an observe-reflect-replan loop for long-form audio-visual reasoning [2605.28192].

**Reasoning-trace distillation and prompt-structured inference** occupy an intermediate position between pure prompting and full architectural redesign. SDRT uses a dual-query reasoning procedure to derive reasoning-guided responses, followed by self-distillation with an intervention adapter, a cross-modal skip connection, and an ensemble of diverse reasoning traces [2503.01754]. CMR-SPB introduces ECV prompting—Extract, Connect, Verify—motivated by the observation that entity linking is the primary bottleneck [2508.16198].

These families are not mutually exclusive. Recent systems increasingly combine graph or memory structures, retrieval and tool use, and structured prompting or policy learning.

## 4. Multi-hop mechanisms and cross-modal evidence integration

At the algorithmic level, CMR systems differ most sharply in how they represent state, propagate information, and control reasoning depth.

In graph neural approaches, multi-hop reasoning is implemented through message passing. WebQA source retrieval uses standard aggregation,
\[
x'_i = W_1 x_i + W_2 \cdot \text{mean}_{j \in \mathcal{N}(i)} x_j
\]
or gated propagation,
\[
\eta_{i,j} = \sigma(W_3 x_i + W_4 x_j), \qquad
x'_i = W_1 x_i + \sum_{j \in \mathcal{N}(i)} \eta_{i,j} W_2 x_j
\]
where the gate $\eta_{i,j} \in [0,1]$ is learned as a function of both node embeddings [2501.04173]. The key claim of that work is that both graph structure and adjacency matrix are task-related prior knowledge, and graph structure can be leveraged to improve retrieval performance [2501.04173].

In reinforcement-learning formulations, multi-hop reasoning is cast as sequential decision making. MMKGR defines states around the current entity, query, and neighborhood, and rewards the agent with a 3D reward composed of destination reward, distance reward, and diverse reward to address sparse reward and encourage short, diverse, successful paths [2209.01416]. CMR-Agent similarly uses an iterative MDP for 2D–3D registration, with state $\mathcal{S}^k$, action $T^k \in SE(3)$, and reward based on improvement in point-to-point alignment [2408.02394]. Its hybrid state representation combines 2D projected features and 3D frustum-alignment cues:
\[
\mathcal{S}^k = \text{Cat}(\psi_s(\mathcal{S}^k_{2d}), \phi_s(\mathcal{S}^k_{3d}))
\]
to address neutral states caused by the spatial truncation of camera frustum [2408.02394].

In retrieval-augmented multimodal LLM systems, the hop mechanism is explicit in tool invocation. MMhops-R1 defines a trajectory
\[
\tau = \{(s_0, a_0, o_1), (s_1, a_1, o_2), \dots, (s_T, a_T)\}
\]
and a reward
\[
R(\tau) = \alpha \cdot R_{\text{outcome}}(\tau) + \beta \cdot R_{\text{format}}(\tau) + \gamma \cdot R_{\text{action}}(\tau)
\]
so that planning, retrieval, and synthesis are jointly optimized [2512.13573]. AOP-Agent uses a planner, observation tools, reflector, and reasoner, supported by working and evidence memories, to perform active perception over sparse and temporally dispersed audio-visual evidence [2605.28192].

Path-based evaluation has clarified that extraction alone is not the main challenge. CMR-SPB reports that on tasks solvable using just one modality, or all given but only one relevant, models perform near-perfectly (>90% accuracy); the struggle is in integrative multi-hop reasoning [2508.16198]. Its step-by-step analysis shows that most failures occur in connecting two entities rather than in end-stage attribute extraction [2508.16198]. FCMR reaches a similar conclusion in a different domain: the critical bottleneck is the information retrieval phase, with models often knowing which modality to use but failing to correctly extract precise details from charts or complex tables [2412.12567]. CRIT’s manual error analysis identifies evidence localization error as the most frequent category, at 55% of audited GPT-4o mistakes [2604.01634].

This convergence across benchmarks suggests that the main systems problem in CMR is often not modality encoding in isolation, but faithful localization and chaining of complementary evidence under path constraints.

## 5. Empirical findings and comparative results

Reported results consistently show that CMR remains far from solved, especially as hop count, path complexity, or modality diversity increases.

On WebQA source retrieval, the VLP-VinVL transformer baseline achieves 68.1 on image, 69.5 on text, and 68.9 combined F1, whereas graph-based methods achieve lower absolute scores but remain competitive: Dense SuperNode 62.8 combined, Star Graph 63.1, SuperNode Gated Graph 54.5, and Star Gated Graph 63.4 [2501.04173]. The paper also states that the proposed method shows a performance gain of **4.6% retrieval F1score over the transformer baselines**, despite being a very light model [2501.04173]. Since the details section reports the main comparison table with graph models below VLP-VinVL on the combined split, this apparent discrepancy should be read cautiously; a plausible interpretation is that the 4.6% gain refers to a different retrieval setting or baseline configuration discussed elsewhere in the paper [2501.04173].

MultiModalQA demonstrates the value of explicit multi-hop modeling. On the test set, AutoRouting obtains 38.2 F1 on multi-modal questions, while ImplicitDecomp reaches 51.7; human performance is 90.1 [2104.06039]. The question type classifier achieves 91.5% accuracy, and for compositional questions, correct first-hop predictions substantially improve second-hop performance [2104.06039].

In knowledge graph reasoning, MMKGR reports strong gains over prior approaches on WN9-IMG-TXT and FB-IMG-TXT, with MRR 80.2 and 71.3 respectively, compared with 62.4 and 50.6 for RLH [2209.01416]. RConE reports dramatic gains on sub-entity queries and slight trade-offs on classic entity queries while remaining on par or better than BetaE and only moderately lower than ConE [2408.11526].

Recent multimodal LLM evaluations are markedly more pessimistic. In MMCV open-book claim verification, GPT-4o, Gemini, and LLaVA show declining F1 as hop count increases; for example, Gemini achieves 79.20 on 1-hop, 71.66 on 2-hop, 65.86 on 3-hop, and 66.97 on 4-hop, while human annotators remain far ahead, around 83–85% on 4-hop [2411.09547]. FCMR reports that proprietary MLLMs are markedly higher on Easy but only 30.4% on Hard for Claude 3.5 Sonnet, with random selection at approximately 12.3% [2412.12567]. In M$^3$-Med, humans reach approximately 0.80 mIoU, while the best models remain below 0.50 for complex questions [2507.04289]. In MOV-Bench, direct inference with Qwen3-Omni-Instruct scores 52.8%, while AOP-Agent raises this to 62.6%, and on long videos from 45.5% to 60.9% [2605.28192].

Training on targeted CMR data has shown substantial transfer benefits. CRIT reports that Qwen2.5-VL-7B improves from F1 29.1 to 59.5 on natural images, from 27.8 to 42.2 on video frames, and from 9.6 to 22.5 on scientific papers after CRIT training [2604.01634]. Adding CRIT to the training mix raises SPIQA METEOR from 3.60 to 10.53, MMQA EM from 27.3 to 30.0, VEGA ROUGE-L from 29.5 to 35.1, and FCMR F1 from 44.9 to 50.5 [2604.01634].

These findings support a consistent picture: narrow tasks with explicit structure can be improved substantially by graph or multi-hop modeling, but robust general-purpose CMR across realistic modalities remains challenging even for strong multimodal LLMs.

## 6. Interpretability, reliability, and open problems

Interpretability is a recurring concern because multi-hop chains amplify the cost of hidden failure. The survey on interpretable cross-modal reasoning organizes explanation methods into visual, textual, graph, symbol, and multimodal explanation, emphasizing that multi-hop interpretability enables tracing outputs to specific steps and evidence chains [2309.01955]. Graph and program approaches are especially amenable to explicit hop-level explanations, while chain-of-thought-style rationales make reasoning sequences legible to end users [2309.01955].

Reliability problems appear in several forms. The LLM survey identifies hallucination and reliability, interpretability, modality scalability, and computational cost as core limitations of LLM-based CMR systems [2409.18996]. MMCV reports overconfidence, visual misinterpretation, temporal and logical reasoning failures, and hallucination risks in both modeling and dataset construction [2411.09547]. FCMR shows that models often default to false when uncertain, struggle particularly with charts and with negative-number comparisons, and may skip required modalities [2412.12567]. M$^3$-Med notes malformed output and refusal answers in general-purpose multimodal LLMs [2507.04289].

Benchmark design itself is now treated as a research problem. CMR-SPB argues that biased benchmarks risk misrepresenting model performance and proposes the Path Balance Score
\[
\text{PBS} = \frac{\mu_{\text{path}}}{1 + \sigma_{\text{path}}}
\]
to combine mean accuracy with variance across reasoning paths [2508.16198]. CRIT argues that most existing multimodal benchmarks fail to capture true multi-hop cross-modal ability because questions are often solvable using a single modality alone [2604.01634]. FCMR similarly emphasizes contamination resistance and strict enforcement of three-hop cross-modal reasoning [2412.12567].

Several forward directions recur across papers. These include richer graph structures and edge types [2501.04173], better encoders and cross-modal representations [2501.04173], improved chart/table/text grounding [2412.12567], robust cross-modal linkage skills rather than only extraction [2508.16198], comprehensive and path-balanced benchmarks as a new gold standard [2508.16198], and training methods that instill dynamic planning and retrieval policies [2512.13573]. Agentic active-perception work further suggests that iterative observe–reflect–replan loops may be necessary when evidence is sparse, temporally dispersed, and distributed across modalities [2605.28192].

A broad synthesis of the literature is that CMR should not be reduced to multimodal fusion. The accumulated evidence indicates that its central technical difficulty lies in enforcing faithful intermediate structure: locating evidence, connecting entities across modalities, controlling hop sequences, and preserving grounding as reasoning depth increases. This suggests that future progress will depend as much on structured evaluation and controllable reasoning procedures as on larger foundation models alone.

Source: https://www.emergentmind.com/topics/cross-modal-multi-hop-reasoning-cmr