InfoCausalQA: Multimodal Causal Reasoning

• The paper introduces InfoCausalQA, a benchmark combining structured visual data and text to assess implicit causal inference from infographics.
• It defines two tasks: quantitative reasoning via numerical trend projections and semantic reasoning over five distinct causal relation types.
• Empirical results reveal that current vision-language models barely exceed chance on numerical tasks, while performing relatively better on effect-based causal questions.

Searching arXiv for the benchmark and closely related causal QA work. I’m checking arXiv for "InfoCausalQA" and related causal QA benchmarks. InfoCausalQA is a multiple-choice benchmark for evaluating non-explicit causal inference from infographics, defined as images that combine structured visual data with textual context. It was introduced to test whether vision-LLMs can move beyond perception and surface matching to perform causal reasoning grounded in infographic structure, numerical trends, and accompanying explanation text. The benchmark contains 1,482 questions built from 494 real-world infographic–text pairs and is organized into two tasks: quantitative causal reasoning from inferred numerical trends, and semantic causal reasoning over five relation types—Cause, Effect, Intervention, Counterfactual, and Temporal (Ka et al., 8 Aug 2025).

1. Benchmark definition and problem setting

InfoCausalQA targets causal reasoning in a multimodal setting where the causal relation is not explicitly stated in a single sentence and must instead be inferred from chart structure, quantitative patterns, and explanatory text. Task 1 focuses on quantitative causal reasoning. Its questions are framed as four-option multiple-choice problems in which exactly one option is correct, and the model must answer “what if…” or “if trend $X$ continues…” queries that require projecting numerical trends and inferring implied causal dynamics. One example given for this task is: “If the line showing average income continues to rise at its current slope, which region’s income gap will narrow the fastest?” (Ka et al., 8 Aug 2025)

Task 2 focuses on semantic causal reasoning. Here the input consists of an infographic $I$, its accompanying text $T$, and a question prompt $Q$. Each question presents four options, and there may be more than one correct answer. The benchmark treats this as a multi-label prediction problem over the option set $\{A,B,C,D\}$, with relation types spanning Cause, Effect, Intervention, Counterfactual, and Temporal. Example prompts include a Cause question—“Which of the following most likely explains the sudden drop in metro ridership after 2020?”—and a Counterfactual question—“If road transport had accounted for only 1 % of food-mile emissions, which statements would change?” (Ka et al., 8 Aug 2025)

The formalization follows the benchmark specification. For Task 1, a question is a tuple $(I, q_1,\ldots,q_4)$, and the model predicts

$$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$

Accuracy is then measured over $N=494$ questions. For Task 2, the benchmark defines $X=(I,T,Q)$ and factorizes prediction across options:

$P(\hat Y \mid X)=\prod_{j=1}^4 P(\hat y_j \mid X),$

where $I$0 indicates whether option $I$1 is selected (Ka et al., 8 Aug 2025).

2. Data sources and construction pipeline

The dataset was manually collected from four public sources: Gallup, Our World in Data, Pew Research Center, and PPIC. In total, 494 infographic–text pairs were scraped and hand-curated. For Task 1, only the infographic image $I$2 is used. For Task 2, the benchmark additionally provides the accompanying explanatory paragraph $I$3 (Ka et al., 8 Aug 2025).

Question generation was performed with GPT-4o using prompt templates. For Task 1, GPT-4o generated one multiple-choice question per infographic, with four options and one correct answer, specifically embedding numerical “if…then…” causal scenarios. For Task 2, GPT-4o generated two multiple-choice questions per infographic, each tied to two distinct causal types selected from the five-type inventory, with four options of which at least two are correct. All prompts used fixed temperature $I$4 to ensure logical consistency (Ka et al., 8 Aug 2025).

Human revision constituted a central part of the pipeline. The revision process enforced three constraints. First, under Implicit Visual Grounding, any direct restatement of chart values was rewritten into a more implicit description, such as “the line rises steeply” rather than “the line goes from 10 to 20.” Second, under Logical Consistency, ill-formed or factually misaligned questions were corrected or replaced. Third, under Causality-type consistency, Task 2 questions were audited to ensure that each item matched its intended relation type; for example, a Counterfactual item had to genuinely ask what would happen if some condition had been different (Ka et al., 8 Aug 2025).

This construction protocol is important because the benchmark explicitly aims to prevent answering by surface-level cue matching alone. A plausible implication is that InfoCausalQA is designed not merely as a visual reading-comprehension set, but as a stress test for inference over structured multimodal evidence.

3. Task design and evaluation protocol

The benchmark uses a zero-shot evaluation setting with no fine-tuning. The evaluated closed-source models are o1, GPT-4o, and Claude 4 Sonnet. The evaluated open-source models are InternVL3-38B-Instruct, InternVL-2.5-MPO (26B and 8B), Qwen2.5-VL-Instruct (72B, 32B, and 7B), LLaVA-OneVision-7B, Idefics2-8B, and Phi-3.5-Vision-Instruct (Ka et al., 8 Aug 2025).

Task 1 uses all 494 multiple-choice questions for evaluation. Task 2 uses all 988 multiple-choice questions, since each infographic contributes two Task 2 items. The benchmark does not define separate train, validation, and test splits; instead, all questions are used for zero-shot assessment (Ka et al., 8 Aug 2025).

For Task 1, the evaluation metric is standard accuracy:

$I$5

For Task 2, the benchmark adopts Select-All-That-Apply metrics from SATA-Bench. These are Exact Match (EM), Precision, Recall, and Jaccard Index (JI), together with three count-bias measures: Mean Count Difference (CtDif), Mean Absolute Count Difference (CtDifAbs), and Count Accuracy (CtAcc). The Task 2 definitions are:

$I$6

$I$7

$I$8

The count-bias measures are

$I$9

$T$0

The paper does not report significance testing (Ka et al., 8 Aug 2025).

4. Empirical results and error profile

On Task 1, quantitative causal reasoning remains difficult. Claude 4 Sonnet obtains $T$1 accuracy, o1 $T$2, GPT-4o $T$3, and Qwen2.5-VL-72B-Instruct also $T$4. InternVL3-38B-Instruct scores $T$5, and other models range from $T$6 to $T$7. The paper notes that even the best model “barely exceeds chance (25%).” Open-source Qwen2.5-VL-72B matches GPT-4o, whereas smaller models such as Idefics2-8B fall below $T$8 (Ka et al., 8 Aug 2025).

On Task 2, the strongest reported Exact Match is achieved by o1, with $T$9, $Q$0, $Q$1, $Q$2, $Q$3, $Q$4, and $Q$5. GPT-4o records $Q$6, $Q$7, $Q$8, $Q$9, $\{A,B,C,D\}$0, $\{A,B,C,D\}$1, and $\{A,B,C,D\}$2. Claude 4 Sonnet scores $\{A,B,C,D\}$3, while Qwen2.5-VL-72B reaches $\{A,B,C,D\}$4, $\{A,B,C,D\}$5, and $\{A,B,C,D\}$6 (Ka et al., 8 Aug 2025).

The type-specific breakdown shows a stable asymmetry across models. For o1, Exact Match is $\{A,B,C,D\}$7 on Effect, $\{A,B,C,D\}$8 on Cause, $\{A,B,C,D\}$9 on Intervention, $(I, q_1,\ldots,q_4)$0 on Counterfactual, and $(I, q_1,\ldots,q_4)$1 on Temporal. GPT-4o shows the same pattern, with $(I, q_1,\ldots,q_4)$2 on Effect and $(I, q_1,\ldots,q_4)$3 on Temporal; Qwen2.5-72B similarly reaches $(I, q_1,\ldots,q_4)$4 on Effect and $(I, q_1,\ldots,q_4)$5 on Temporal. The paper states that all models do best on Effect questions and worst on Temporal reasoning (Ka et al., 8 Aug 2025).

The qualitative analysis attributes errors to different failure modes. In a Cause example about low fertility rates, o1 and GPT-4o correctly integrated background knowledge and chart cues, whereas a smaller model refused to answer and over-relied on explicit evidence. In a Counterfactual food-mile example, only o1 fully succeeded; GPT-4o made a numeric error and Qwen2.5-72B mis-reasoned about total emissions. The benchmark also reports that pilot human evaluators achieve near-ceiling performance, described as $(I, q_1,\ldots,q_4)$6 accuracy or EM, although exact human scores are not tabulated. This suggests that the benchmark measures a capability gap rather than an annotation ceiling effect (Ka et al., 8 Aug 2025).

5. Position within causal QA and reasoning research

InfoCausalQA occupies a specific place within a broader causal reasoning literature. In text-only causal question analysis, Quriosity identified a substantial proportion of causal questions in naturally occurring human queries—$(I, q_1,\ldots,q_4)$7 out of $(I, q_1,\ldots,q_4)$8, or $(I, q_1,\ldots,q_4)$9—and formalized causal questions through three mechanistic natures and Pearl’s ladder of causation (Ceraolo et al., 2024). That work concerns causal-query identification rather than infographic-grounded reasoning, but it provides a clear account of why causal intent detection matters.

In fine-grained textual QA, the Causal QA portion of the FCR dataset introduced seven relation labels—Cause, Enable, Prevent, Cause_By, Enable_By, Prevent_By, and Irrelevant—and reported $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$0 QA pairs from analyst reports. On that benchmark, RoBERTa-large reached $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$1 and $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$2, while error analysis showed particular difficulty for Enable and Enable_By (Yang et al., 2022). Compared with that setting, InfoCausalQA shifts the challenge from relation extraction in prose to causal inference over charts and infographic context.

ExpliCa isolates another dimension of the problem: explicit causal reasoning marked by connectives such as “so,” “because,” “then,” and “after.” It evaluates whether models can distinguish causal from temporal relations and reports that even top systems struggle to reach $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$3 accuracy, with systematic confusion between causal and temporal cases and a marked dependence on event order (Miliani et al., 21 Feb 2025). InfoCausalQA’s poor temporal results are consistent with that pattern.

The benchmark also belongs to a multimodal lineage. Causal-VidQA introduced Description, Explanation, Prediction, and Counterfactual questions for videos and found that models are substantially weaker on commonsense reasoning than on description, with best reported performance around $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$4 on Prediction and $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$5 on Counterfactual versus human performance near $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$6 mean accuracy (Li et al., 2022). InfoCausalQA extends the multimodal causal agenda from temporal video clips to infographics, where the required reasoning often combines implicit numerical trend analysis with language-conditioned causal interpretation.

6. Architectural implications and future directions

The benchmark paper concludes that current vision-LLMs show limited capability in computational reasoning and even more pronounced limitations in semantic causal reasoning over infographics (Ka et al., 8 Aug 2025). Several adjacent works suggest possible design directions, although these are broader than the benchmark itself.

For retrieval-intensive settings, CAWAI proposes a dense retriever with three encoders—Cause Encoder $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$7, Effect Encoder $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$8, and a frozen Semantic Encoder $$\hat y = \arg\max_{j\in\{1\ldots 4\}} P(y=j \mid I, Q).$$9—trained with causal and semantic objectives. It outperforms semantic retrievers on causal retrieval tasks, especially in large-scale settings, and shows strong zero-shot generalization on scientific QA (Shin et al., 7 Apr 2025). This suggests that infographic QA systems could benefit from retrieval layers that distinguish “retrieve causes” from “retrieve effects” rather than relying on undirected semantic similarity.

For graph-based causal QA, an Actor-Critic approach over a causality graph reformulates binary questions of the form “Does $N=494$0 cause $N=494$1?” as sequential search. The reported agent visits fewer than $N=494$2 nodes per question, compared with over $N=494$3 for breadth-first search, and returns explanatory paths with provenance (Blübaum et al., 2023). That formulation is not infographic-specific, but it illustrates a path toward explicit mechanistic explanations.

For explainability, e-CARE adds free-text causal explanations to $N=494$4 reasoning questions and shows that explanation generation remains difficult even when explanation information helps QA accuracy and stability (Du et al., 2022). More recently, ChainReaction! uses natural-language causal chains as intermediate representations for causal video QA and reports that supplying gold causal chains allows accuracy near $N=494$5 across three datasets, while human studies preferred chain-based explanations in $N=494$6 of cases (Parmar et al., 28 Aug 2025). A plausible implication is that future systems for infographic reasoning may require explicit intermediate causal representations rather than monolithic end-to-end prediction.

In that sense, InfoCausalQA functions both as an evaluation benchmark and as a diagnostic instrument. Its strongest empirical signal is not that current models fail uniformly, but that they fail selectively: numerical trend projection is weak, semantic causal inference is stronger but still below human performance, Effect questions are easier than Cause or Counterfactual in many cases, and Temporal reasoning is consistently the hardest (Ka et al., 8 Aug 2025). That profile makes the benchmark a useful reference point for work on multimodal causal inference, structured reasoning, and explanation-grounded evaluation.