M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

Abstract: We present M$^3$-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal LLMs (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M$^3$-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M$^3$-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.

• The paper introduces M$^3$-VQA, a benchmark designed to evaluate multimodal large language models' (MLLMs) ability to perform complex reasoning across multiple entities and inference steps.
• M$^3$-VQA formalizes visual question answering tasks demanding intricate reasoning over intertwined visual and textual information, with tasks needing up to four reasoning hops.
• Analysis of M$^3$-VQA shows cutting-edge models struggle significantly without explicit evidence, identifying the necessity of advanced retrieval and reasoning methods.

M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

Introduction and Motivation

M$^3$-VQA is introduced as a unified and rigorous benchmark to assess the fine-grained multimodal entity recognition and multi-hop reasoning capacity of Multimodal LLMs (MLLMs). Existing knowledge-based VQA datasets are primarily limited by their focus on single-entity, low-complexity questions and coarse-grained scenarios. Contemporary visual reasoning demands more: identifying multiple, diverse entities across both visual and textual modalities, retrieving relevant multimodal knowledge, and performing both sequential and parallel reasoning over multiple hops. M$^3$-VQA is engineered to fill these gaps by supporting evaluations that closely parallel the complexity and diversity encountered in real-world multimodal understanding.

Task and Dataset Design

M$^3$-VQA formalizes the VQA task as $f: (I, Q, K) \rightarrow A$, with $I$ as the image, $Q$ as the complex multi-entity question, $K$ as an explicit multimodal knowledge base (Wikipedia), and $A$ as the answer. Central to the benchmark are two core task dimensions: entity complexity (number/diversity of entities involved) and hop complexity (number of reasoning steps necessary for the evidence chain).

Two primary reasoning patterns, both instantiated with explicit supporting evidence, structure the dataset:

• Parallel Multi-Hop Reasoning: Entities are reasoned about independently then their attributes are aggregated to produce the final answer.
• Sequential Multi-Hop Reasoning: Downstream reasoning hops depend on upstream inference, necessitating chained evidence retrieval.

  Figure 1: Examples of parallel and sequential multi-hop reasoning in M$^3$-VQA.

The construction leverages and extends existing fine-grained image datasets (e.g., CelebTo, FGVD) and knowledge-based VQA datasets (e.g., EVQA, InfoSeek), using explicit Wikidata entity linking and Wikipedia-sourced multimodal grounding. Question generation for both paradigm types is facilitated by a combination of template-based methods, SPARQL query-driven answer extraction, and LLM-based paraphrasing/extension.

Dataset Analysis

M$^3$0-VQA comprises 13,125 (image, question, answer) triples, covering 10,565 unique images and 7,611 unique questions. Entity coverage spans persons, vehicles, architectural sites, brands/logos, foods, animals, plants, and landmarks, with annotated links to Wikidata and Wikipedia for each entity. The task complexity is calibrated: 32.8% of questions require four or more hops, and 32.8% require reasoning over four or more entities.

Figure 2: Complexity of M$^3$1-VQA.

Figure 3: Types of questions covered in M$^3$2-VQA.

Figure 4: Types of data in M$^3$3-VQA.

Figure 5: Examples of M$^3$4-VQA questions across diverse fine-grained entity types.

All questions are supported by traceable gold evidence, extracted and annotated at the sentence level from Wikipedia. Evaluation metrics are built on intersection-over-union (IoU) between predicted and gold answer sets, with partial credit for nearly-correct completions.

Experimental Protocols

The evaluation paradigm is stratified into three principal settings:

• Original: The model receives only the question and image.
• Oracle: The model is provided with gold evidence at varying granularity (sentence, section, entity name).
• Knowledge Base (KB): The model must retrieve supporting evidence from a provided knowledge base, via either heuristic (single query) or agentic (decomposed, planning-driven) retrieval.

Model Performance and Analysis

Extensive experiments on 16 SOTA MLLMs reveal pronounced deficiencies:

• Even the best-performing model (Qwen2.5-VL-72B-Instruct) achieves only 32.6% accuracy in the Original setting, demonstrating inadequate background knowledge acquisition and entity reasoning in the absence of supporting evidence.
• Access to gold sentence-level evidence increases SoTA accuracy to 58.7% (InternVL2.5-78B), quantifying the persistent reliance of MLLMs on precise external context for nontrivial reasoning.

  Figure 6: Model accuracy across different hop and entity counts, under the sentence evidence setting.

• Parallel and sequential multi-hop questions incur higher error rates as the number of entities or hops increases, underlining scaling limitations of current model architectures with respect to compositionality and reasoning depth.
• In KB settings, naïve (heuristic) retrieval results in significantly lower accuracy compared to agentic, step-wise retrieval frameworks.
• Agentic retrieval outperforms heuristic methods by up to 6% absolute, highlighting the necessity of decompositional query planning and structured evidence gathering for complex cases.

  Figure 7: Schematic diagram and case study of the agentic retrieval model.

Qualitative Case Studies

The dataset supports the evaluation of multi-hop reasoning over both visual and textual entities. Case studies illustrate the requirements for explicit object detection, fused entity recognition, stepwise retrieval (potentially across several Wikipedia documents), and robust answer aggregation.

Figure 8: Four-hop multi-entity vehicle manufacturer aggregation, with explicit entity linking and document evidence.

Figure 9: Three-hop sequential geographical and culinary reasoning, showing the grounding chain from landmark to local dish to species assessment.

These examples consistently demonstrate that single-shot retrieval or end-to-end reasoning pipelines are structurally inadequate in the face of multi-hop, multi-entity compositionality.

Implications and Future Directions

M$^3$5-VQA redefines the complexity bar for VQA reasoning:

• It exposes the brittle nature of current MLLM reasoning chains without access to externally retrieved, fine-grained evidence.
• Evaluation under the KB setting reveals significant headroom for improvements in both multi-hop, multi-entity retrieval and subsequent multi-step reasoning, independent of model scale.
• The strong effectiveness of agentic retrieval highlights the need for architectures that integrate stateful query decomposition, grounded multimodal perception, and iterative evidence synthesis.
• Practically, models succeeding on M$^3$6-VQA would be more reliable for open-domain QA, web-based search/grounded agents, and other settings where fine-grained, explainable visual understanding is mission-critical.

Theoretically, this work re-centers VQA toward the open-domain QA paradigm, demanding near-perfect entity detection, world-knowledge retrieval, and the capacity for explicit multistep reasoning with transparent attribution.

Conclusion

M$^3$7-VQA advances multimodal VQA benchmarks by enforcing challenging evaluation on fine-grained, multi-entity, multi-hop inference with explicit, traceable evidence chains. Neither model scaling nor basic retrieval strategies suffice; agentic retrieval and compositional reasoning become necessary. This benchmark sets the stage for future work on entity-centric multimodal alignment, retrieval-augmented generation, and agentic multimodal systems capable of effectively traversing the full chain from grounded perception to logical conclusion.

Citation: "M$^3$8-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering" (2604.25122)