ReasonVQA: Multi-Hop & Rationale VQA

• ReasonVQA is a visual question answering paradigm that links images and videos to encyclopedic knowledge and human rationales, enabling explicit multi-hop and temporal reasoning.
• It includes a large benchmark with over 4 million questions generated via recursive templates over Wikidata, emphasizing scalable structural knowledge and hop complexity.
• Additionally, the POVQA variant provides a compact movie-domain corpus with dense human rationale annotations to improve answer clarity and explanation faithfulness.

ReasonVQA is a name used in recent visual question answering literature for two distinct but closely related resources. In one usage, it denotes a large-scale benchmark that links real images to structured encyclopedic knowledge in order to generate 1-hop, 2-hop, and 3-hop visual questions over Wikidata (Tran et al., 22 Jul 2025). In another, it denotes a compact movie-domain supervision set of question-answer-reason triples introduced together with POVQA for long-context video question answering with explicit human rationales (Dahal et al., 1 Oct 2025). Across both usages, the term marks a shift from answer-only VQA toward settings in which visual grounding must be coupled to explicit reasoning, external knowledge, temporal evidence, or rationale supervision.

1. Nomenclature and scope

In current usage, “ReasonVQA” does not refer to a single canonical dataset. It refers to two separate resources that emphasize different forms of reasoning.

Usage	Core setting	Reported scale
ReasonVQA	Multi-hop VQA with structural knowledge from Wikidata	598,525 images and 4,174,024 questions
ReasonVQA in POVQA	Long-form movie video QA with human rationales	12 movies and 239 question-answer-reason triples

The first resource, titled “ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering,” is a benchmark-generation framework and dataset designed to test image-grounded retrieval and composition of encyclopedic facts (Tran et al., 22 Jul 2025). The second is the rationale-annotated dataset used by “POVQA: Preference-Optimized Video Question Answering with Rationales for Data Efficiency,” where the term denotes a compact movie QA corpus intended for data-efficient adaptation of large vision-LLMs to long-context, reasoning-grounded video QA (Dahal et al., 1 Oct 2025).

This terminological overlap suggests two major strands of contemporary reasoning-oriented VQA research. One strand emphasizes scalable knowledge integration and explicit hop complexity; the other emphasizes rationale supervision, temporal summarization, and faithfulness in long-video settings.

2. ReasonVQA as a structural-knowledge benchmark

The benchmark introduced under the title “ReasonVQA” is built around the claim that many VQA questions cannot be answered from pixels alone. Its construction pipeline links real images to Wikidata through object/entity grounding, retrieves structured knowledge with SPARQL, and generates questions by recursive template composition over knowledge-graph paths (Tran et al., 22 Jul 2025).

Image sources are Visual Genome and Google Landmark Dataset v2. For Visual Genome, WordNet-linked object annotations are converted into WordNet synset IDs and then mapped to Wikidata entities. For GLDv2, Wikimedia URLs are used to identify landmark entities. Once a root concept is established, the system traverses surrounding Wikidata triples and generates 1-hop, 2-hop, or 3-hop questions. One-hop questions retrieve a single property of the linked entity. Multi-hop questions are formed by nesting a main template with one or more sub-clause templates. The framework reports 182 main templates, 100 sub-clause templates, and 182 distinct properties.

The knowledge source is explicitly structural rather than commonsense-only. Questions therefore target properties such as architect, author, country, capital, official language, mother, place of birth, height, and width. The framework also allows Visual Genome scene graphs to contribute visually contextualized sub-clauses, so some questions combine scene-level relations with external knowledge rather than relying on the knowledge graph alone.

The dataset is organized into 20 domains, including places/locations, person/institutions, temporal concepts, characteristics/properties, language/cultural, history/events, physical geography, politics, economics, nature, technology, science, health, education, art, philosophy/spiritual beliefs, media, environment, law, and food. Answer candidates for multiple-choice settings are generated differently for fixed, date, number, and literal answer types. For numeric distractors, the reported range is

$[m,n]$

with

$m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$

where $i$ is the correct answer and $N=3$ is the number of false choices.

Scale is one of the benchmark’s defining properties. The full dataset contains 598,525 images and 4,174,024 questions, with 1,358,634 1-hop questions, 2,809,960 2-hop questions, and 5,430 3-hop questions. It also reports 123,204 unique questions, 73,068 unique answers, 123,411 unique choices, an average question length of 9.77 words, and an average answer length of 1.53 words. Two evaluation subsets are also reported: ReasonVQA-U, with 13,326 images and 78,007 questions, and ReasonVQA-B, with 8.7K images and 51.9K questions.

The benchmark includes a balancing procedure intended to reduce answer-frequency bias. Questions are grouped by property, the most frequent answers are iteratively downsampled, and image-level splitting preserves answer-distribution structure under a 70% train and 30% test regime. A user study on 1000 random image-question pairs with 20 participants found 58.1% of questions rated natural and 25.8% rated very natural; 2.5% were marked as having grammatical errors. That profile indicates a typical tradeoff of large template-based generation: scale and controllable reasoning depth are obtained at the cost of some linguistic variability.

3. ReasonVQA as a movie-domain rationale corpus in POVQA

In POVQA, ReasonVQA refers to a small but densely supervised benchmark for long-form movie video question answering, introduced as the supervision resource for a data-efficient LVLM adaptation pipeline (Dahal et al., 1 Oct 2025). The dataset contains 239 question-answer-reason triples drawn from 12 movies spanning 12 genres: romance, historical, biography, western, fantasy, action, mystery, thriller, animation, drama, sci-fi, and documentary. The eval set specifically contains sci-fi and western titles, and the source material collectively covers over 1 million raw frames.

The annotation format is deliberately explanation-centric. Each example contains a question, a short final answer, and a natural-language rationale explaining how the answer is justified from visual and subtitle context. The stated goal is not only correct answering but also faithfulness to in-context evidence and reduced drift toward unsupported external knowledge. This makes ReasonVQA, in the POVQA sense, a supervision resource for answer generation plus rationale generation rather than merely a benchmark for answer accuracy.

POVQA’s task formulation begins from a video clip

$\mathcal{V}=\{I_t\}_{t=1}^{T}$

sampled at $f$ fps and subtitle spans

$\mathcal{S}=\{(a_j,b_j,\text{text}_j)\}_{j=1}^{J}.$

Instead of feeding raw frames directly, each second is compressed into one pooled image at 1 Hz. If

$S=\lfloor T/f \rfloor,$

then second $s$ is summarized by an average image

$$\bar{I}_s = \sum_{\tau\in\mathcal{W}_s} w_s(\tau)\, I_\tau,$$

with four pooling operators studied: Weighted Average (WA), Weighted Average Exponential (WAE), Weighted Average Ramp (WAR), and Blend–Blur with Last Frame (BBLF). Subtitles are aligned to seconds and interleaved with pooled images to form the multimodal sequence

$m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$0

The base model is Qwen2.5-VL-7B. Supervision is explicitly two-turn: the model first produces a rationale under a “Reasoning:” header and then a short answer under a “Final Answer:” header, with targets denoted by $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$1 and $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$2. Training uses Supervised Fine-Tuning (SFT) with QLoRA, followed optionally by Direct Preference Optimization (DPO) over preferred versus dispreferred rationale-plus-answer sequences. The reported training budget is lightweight by LVLM standards: 2 epochs over 239 annotated examples on one NVIDIA A40-48Q GPU, with QLoRA rank/$m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$3, dropout $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$4, seed $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$5, gradient accumulation $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$6, learning rate $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$7 for SFT and $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$8 for DPO.

A major methodological claim is token efficiency. For a worked example using a 5-minute clip at 24 fps, capped to $m=\frac{i}{2}, \quad n=max(1.5i, m+2N),$9 seconds, with $i$0, subtitle token average $i$1, and $i$2, the paper reports a context length of 16,088 tokens at 1 Hz versus 369,368 if all 24 fps frames were used over the same 60-second cap, or about a 23× reduction.

4. Evaluation regimes and empirical findings

The two ReasonVQA resources are evaluated very differently, reflecting their distinct research aims (Tran et al., 22 Jul 2025, Dahal et al., 1 Oct 2025).

For the structural-knowledge benchmark, evaluation is done in both open-ended and multiple-choice modes. Open-ended scoring uses exact match, substring, and semantic similarity, with semantic similarity computed using all-MiniLM-L6-v2. Multiple-choice prompting requires output of a single letter. On open-ended semantic-similarity accuracy, the benchmark reports the following pairs on ReasonVQA-U / ReasonVQA-B: GPT-4o 62.8 / 60.8, Qwen2.5-VL 59.3 / 58.1, Idefics2 50.8 / 49.7, LLaVA-OV 50.5 / 50.6, and BLIP-2 46.4 / 46.1. In multiple-choice semantic-similarity accuracy, mPLUG-Owl3 reaches 68.9 / 68.1, Mantis-Idefics2 reaches 68.7 / 68.5, and Mantis-SigLIP reaches 67.8 / 67.6. Fine-tuning with LoRA produces large gains for some models, including PaliGemma-2-3B-Mix, which goes from 40.1 zero-shot to 66.8 fine-tuned, and PaliGemma-2-10B-Mix, which goes from 65.6 to 74.5. Performance drops sharply on 3-hop questions, and questions generated with scene graphs are reported as harder than those without them.

For the movie-domain ReasonVQA in POVQA, evaluation separates answer quality from rationale quality. Answer metrics are F1, BLEU-1, BLEU-4 with brevity penalty, ROUGE-L, and embedding cosine similarity; rationale metrics are ROUGE-L-R and Embed Cosine-R. Before fine-tuning, the best pooled baseline reaches only F1 0.212 with BBLF, BLEU-4 0.031 with WA, and ROUGE-L 0.196 with BBLF. After SFT, the best answer F1 reaches 0.550 under WA evaluation with a WAE-trained model, and 0.543 on BBLF evaluation with the same training choice; the highest reported rationale embedding cosine under SFT is 0.597 on WAR evaluation with a BBLF-trained model. After SFT + DPO, answer metrics remain roughly similar, with the paper explicitly describing the incremental gain over SFT as nominal, while rationale quality is often modestly sharpened. The dataset also supports a key-frame-only ablation: lexical answer metrics there can appear competitive, with best F1 0.563, BLEU-4 0.291, and ROUGE-L 0.528, but the paper argues that pooled temporal evidence primarily improves semantic metrics, including +0.013 in answer embedding cosine and +0.033 in rationale embedding cosine over the best key-frame baseline. For zero-shot transfer to TVQA, SFT + DPO reaches 64.7% on a random 5k subset of the TVQA eval split, while a pooling-only zero-shot baseline reaches 69.7% and key-frame-only reaches 56.8%.

Taken together, the empirical profile is bifurcated. The structural-knowledge ReasonVQA stresses knowledge retrieval, hop composition, and dataset scale; the movie-domain ReasonVQA stresses low-data adaptation, long-context summarization, and rationale faithfulness. Both are reported as difficult for current systems, but they are difficult in substantially different ways.

5. Relation to adjacent reasoning-VQA paradigms

ReasonVQA sits within a broader shift from answer-only VQA toward models and benchmarks that expose or supervise intermediate reasoning traces. Several neighboring lines of work make that transition explicit.

LRTA reformulates VQA as full answer generation with natural-language justification and uses a modular “Look, Read, Think, Answer” pipeline: scene graph construction with DETR, question parsing into reasoning instructions, recurrent neural-symbolic execution over the graph, and transformer decoding of full answers (Liang et al., 2020). “Interpretable Visual Question Answering via Reasoning Supervision” uses rationale text available during training to align question-guided and rationale-guided visual attention, treating rationale-induced attention as an implicit grounding teacher (Parelli et al., 2023). “Generating Rationales in Visual Question Answering” conditions GPT-2 rationale generation on ViLBERT answer representations and reports improvements in both rationale quality and VQA accuracy under multitask training (Ayyubi et al., 2020). ReasVQA extends process supervision to VideoQA by generating rationales with an external MLLM, refining them heuristically, and learning with a multitask objective

$i$3

showing that imperfect reasoning traces can still improve NExT-QA, STAR, and IntentQA when treated as auxiliary supervision (Liang et al., 23 Jan 2025).

A second neighboring line targets the validity conditions of questions themselves. “The Promise of Premise” treats a question as relevant only if all of its implied premises hold in the image, constructs the QRPE dataset for Question Relevance Prediction and Explanation, and shows that premise-aware models outperform premise-agnostic ones on question relevance detection (Mahendru et al., 2017). This is especially pertinent to ReasonVQA because many failure modes in knowledge-heavy or rationale-heavy settings are premise failures rather than answer-generation failures.

A third line emphasizes explicit structure or external knowledge. The PSL-based reasoning layer in “Explicit Reasoning over End-to-End Neural Architectures for VQA” reasons over visual relations, semantic parses, and background knowledge from word2vec and ConceptNet, while exposing key evidential predicates (Aditya et al., 2018). REXUP combines an image object-oriented branch, a scene graph-oriented branch, and recurrent “REason, EXtract, UPdate” cells to model step-by-step reasoning in GQA (Luo et al., 2020). This suggests that ReasonVQA, in both senses, belongs to a larger family of work that treats visual reasoning as a problem of structured decomposition rather than of monolithic multimodal fusion alone.

6. Limitations, ambiguity, and significance

Both resources named ReasonVQA are explicitly valuable, but each comes with narrowness as well as strength.

For the structural-knowledge benchmark, the main strengths are scale, explicit hop control, and low-cost automatic generation. The main caveats are equally clear: question generation is template-based; 2.5% of sampled items were marked as having grammatical errors; 3-hop generation is limited because further nesting degrades grammatical quality; and correctness relies on automatic entity linking and Wikidata retrieval rather than large-scale manual factual verification (Tran et al., 22 Jul 2025). For the movie-domain ReasonVQA, the principal strength is dense human rationale annotation over long-context movie video QA, but the dataset is small, subtitle alignment is a dependency, and the paper reports that DPO gains on answer accuracy are limited; in TVQA zero-shot transfer, pooling-only even outperforms SFT+DPO (Dahal et al., 1 Oct 2025).

The name collision itself is consequential. One ReasonVQA is a very large benchmark for image-grounded multi-hop encyclopedic reasoning; the other is a very small but high-value supervision set for temporally extended, explanation-grounded video QA. This suggests that “ReasonVQA” has become less a single dataset identity than a label for a research agenda: VQA systems should be evaluated not only on whether they answer, but also on whether they retrieve the right knowledge, preserve temporal evidence, expose rationales, and remain grounded to the actual input. In that sense, the two resources are complementary rather than redundant. One stresses scalable structural knowledge; the other stresses faithful long-context explanation. Together they define a more demanding conception of visual question answering than answer-only leaderboard evaluation.