VLQA: Multimodal QA Benchmark
- VLQA is a visuo-linguistic dataset of 9,267 image–passage–QA tuples designed to test joint reasoning across visual and textual modalities.
- The dataset’s annotation ensures neither image nor text alone suffices, employing strict quality controls to minimize unimodal shortcuts.
- Baseline evaluations show significant performance gaps compared to human accuracy, highlighting challenges in cross-modal attention and logical inference.
VLQA, short for Visuo-Linguistic Question Answering, is a challenge corpus for question answering over paired image and passage inputs, introduced to evaluate joint inference over image–text modality rather than reasoning in vision and language separately (Sampat et al., 2020). The dataset contains 9,267 image–passage–QA tuples, and each item consists of an image and a reading passage such that the question is designed to combine both visual and textual information; ignoring either modality would make the question unanswerable. The benchmark was proposed in response to the observation that, although strong benchmarks existed for language and vision separately, joint reasoning remained difficult for state-of-the-art computer vision and natural language processing systems.
1. Corpus definition, scale, and modality composition
VLQA comprises 9,267 image–passage–QA tuples organized into standard train, validation, and test partitions (Sampat et al., 2020).
| Split | Size |
|---|---|
| Training | 7,413 |
| Validation | 927 |
| Test | 927 |
The image source space spans three broad types: natural photographs (4,445 items), template-based figures (3,920 items), and free-form diagrams (1,854 items). Template-based figures include bar charts, pie charts, and maps; free-form diagrams include scientific setups and flowcharts. The passages are short, ranging from 1 to 5 sentences, with 34.1 tokens on average, and are drawn from textbooks, encyclopedias, web crawls, or are manually composed to accompany each image.
This composition is central to the benchmark’s intended difficulty. Natural scenes, charts, maps, and scientific diagrams expose different visual primitives, while short reading passages constrain the textual side to compact but information-bearing context. A plausible implication is that VLQA was designed not merely as a generic multimodal dataset, but as a controlled testbed for heterogeneous visual structures paired with concise explanatory text.
2. Annotation protocol and integrity controls
All QA pairs were authored by expert in-house annotators, with two explicit constraints: neither image nor text alone should suffice to answer the question, and exactly one answer in the 2-way or 4-way multiple-choice set should be correct (Sampat et al., 2020). The dataset construction therefore emphasized multimodal dependence at annotation time rather than attempting to impose it only through post hoc filtering.
Quality control proceeded through three stages. First, manual curation discarded image–text pairs that duplicated information. Second, unimodal baselines—question-only, passage-only, and image-only—were run three times with randomized choice order, and any example solved consistently by a single-modality model was removed. Third, annotators attempted to answer each QA item from only one modality; ambiguous or unimodally solvable items were fixed or dropped. Difficulty labels, recorded as easy, medium, or hard, were assigned by majority vote among three raters.
These controls are notable because they target shortcut learning directly. Rather than assuming multimodal necessity from collection heuristics, VLQA explicitly tests for unimodal leakage. This suggests a dataset philosophy closer to adversarial benchmark construction than to passive corpus aggregation.
3. Question taxonomy and answer formats
VLQA organizes questions into three categories (Sampat et al., 2020). The first is purely visual selection, implemented as 2-way or 4-way image multiple-choice questions; an example is “Which image depicts the correct phase of the moon described in the passage?” The second is purely textual, implemented as binary fact-checking; an example is “True or False: All parts of the plant are described in the passage?” The third is integrated visuo-linguistic reasoning, requiring both modalities and appearing as 4-way text multiple-choice and sequencing tasks; an example is “According to the chart and the passage, which city has the highest ratio of green space to population?”
The answer formats are multiple-choice classification with either 2-way or 4-way choice sets. The paper highlights three answer-type labels through examples: 4wI for 4-way image multiple choice, Bin for binary classification, and 4wT for 4-way text multiple choice. Although the benchmark is evaluated as multiple-choice QA, the open-ended answer length is short—1.7 tokens on average—which the paper notes as enabling future use of generative decoders. The combined textual vocabulary across passages, questions, and answer options contains 13,259 distinct tokens.
There is an apparent tension between the global design rule that neither modality alone suffices and the labels “purely visual” and “purely textual.” This suggests that these category names describe the answer presentation format or the local locus of evidence selection, rather than licensing genuinely unimodal solution strategies.
4. Evaluation protocol and baseline performance
The standard evaluation metric in VLQA is exact-match accuracy, defined as the fraction of questions for which the predicted choice matches the gold label (Sampat et al., 2020). The benchmark also distinguishes micro-accuracy and macro-accuracy. Micro-accuracy is the overall accuracy treating every example equally, whereas macro-accuracy is the average of per-category accuracies, such as across answer types or reasoning types, thereby giving equal weight to each category. Although precision, recall, and F1 can be computed by treating each answer choice as a separate class, the standard practice in VLQA is to report overall accuracy.
Unimodal baselines were used for quality control rather than as standard trained VLQA systems. Their reported accuracies were 28.6% for question-only using RoBERTa, 30.2% for passage-only using ALBERT, and 29.5% for image-only using LXMERT. These numbers quantify the residual solvability of the task under single-modality bias and provide a lower reference point for multimodal systems.
Among pre-trained vision–LLMs fine-tuned first on VQA and then on VLQA, LXMERT performed best, with 37.8% on validation and 36.4% on test. The proposed modular method, HOLE (HOpping + Logical Entailment), achieved 40.1% on validation and 39.6% on test, slightly outperforming end-to-end transformer baselines while offering interpretable intermediate outputs. Human performance on the held-out test set reached 84.0%.
| System | Validation accuracy | Test accuracy |
|---|---|---|
| VL-BERT | 34.6% | 35.9% |
| VisualBERT | 34.2% | 33.2% |
| ViLBERT | 35.3% | 34.7% |
| LXMERT | 37.8% | 36.4% |
| HOLE | 40.1% | 39.6% |
| Human | — | 84.0% |
The benchmark gap is therefore substantial: even the modular baseline remains far below expert human performance. The paper’s central empirical claim is that existing vision–language architectures are unable to reason well on VLQA, and that modest gains from HOLE do not close the gap.
5. Error profile and reasoning demands
VLQA exposes several specific reasoning challenges (Sampat et al., 2020). One is cross-modal retrieval, described as “hopping” between image and text to determine which modality contains the relevant clue. Another is multi-step inference: 2,939 of the 9,267 examples require two or more reasoning steps. A third is external knowledge, with 61% of examples demanding commonsense or domain facts beyond the provided context.
The paper’s failure analysis reports 148 human errors, divided into four categories: lack of necessary background knowledge (27/148, 18.2%), misunderstanding of the provided information (47/148, 31.7%), calculation or deduction mistakes (63/148, 42.5%), and ambiguous items (11/148, 7.4%). Even at human level, the dominant error source is not perceptual failure but calculation or deduction. This situates VLQA as a reasoning benchmark rather than a primarily recognition-oriented one.
The reported shortcomings of current models align with these statistics. Pre-trained vision–language transformers struggle to integrate passage information and often ignore one modality. The paper further identifies the lack of structured logical modules for arithmetic, comparison, and temporal-causal chains as a central limitation. In that sense, VLQA targets a regime in which representation learning alone appears insufficient without stronger compositional or symbolic support.
6. Benchmark role, future directions, and acronym ambiguity
VLQA was proposed as a benchmark for reasoning over a visuo-linguistic context, with public dataset, code, and leaderboard availability. The future directions named in the paper are technically specific: better cross-modal attention mechanisms that dynamically choose which modality to attend first; integration of symbolic reasoning or neuro-symbolic modules for arithmetic, counting, and logic; automatic retrieval and incorporation of external knowledge in an end-to-end fashion; extension to more complex diagrams such as multi-panel scientific figures and to longer passages; and diagnostics on linguistic biases together with adversarial sampling to further reduce unimodal shortcuts (Sampat et al., 2020).
A recurrent source of confusion in later literature is that the acronym “VLQA” has been reused for unrelated tasks. In 2024, “VLQA” was used for Video Library Question Answering, defined over archival NASA video assets and an interoperable Retrieval Augmented Generation architecture, but that work explicitly did not release a purpose-built, human-annotated QA dataset with train/validation/test splits (Tevissen et al., 2024). In 2025, “VLQA” was also used for “The First Comprehensive, Large, and High-Quality Vietnamese Dataset for Legal Question Answering,” a distinct legal NLP resource containing 3,129 verified triplets of question, relevant articles, and long-form answer (Nguyen et al., 26 Jul 2025). This suggests that citation disambiguation is necessary when discussing “VLQA” in contemporary scholarship.
Within multimodal reasoning research, however, the 2020 VLQA corpus remains specifically the Visuo-Linguistic Question Answering challenge: a 9,267-item benchmark constructed to ensure that answering requires coordinated use of image and passage, and to expose the gap between current multimodal architectures and human reasoning over mixed visual–textual evidence.