---
title: 'InterChart: Multi-Chart Visual Reasoning Benchmark'
url: https://www.emergentmind.com/topics/interchart
type: topic
---

# InterChart: Multi-Chart Visual Reasoning Benchmark

Searching arXiv for the InterChart paper and closely related chart-understanding work to ground the article in current literature.
InterChart is a diagnostic benchmark for multi-chart visual reasoning that evaluates whether vision-language models can answer questions by connecting information distributed across \(2\)–\(3\) related charts rather than reading a single chart in isolation. It is designed for settings such as scientific reporting, financial analysis, and public policy dashboards, where answers often depend on comparison, alignment, or synthesis across multiple visuals rather than extraction from one figure alone [2508.07630]. The benchmark organizes this problem into three tiers of increasing difficulty—DECAF, SPECTRA, and STORM—and reports that state-of-the-art open and closed-source models exhibit steep accuracy declines as chart complexity increases, while also showing that decomposing multi-entity charts into simpler visual units can improve performance [2508.07630]. In the broader chart-understanding literature, InterChart occupies the multi-chart end of a spectrum that includes single-chart question answering systems such as ChartNet [1911.09375], unified chart derendering and comprehension frameworks such as ChartReader [2304.02173], and comparative chart summarization benchmarks such as ChartDiff [2603.28902].

## 1. Concept and motivation

InterChart targets a specific failure mode in contemporary chart-understanding systems: they are often competent at reasoning over isolated, visually uniform charts, but much weaker when relevant evidence is decomposed across several related figures [2508.07630]. The benchmark is motivated by the observation that many practical chart-analysis tasks require cross-chart integration under conditions of heterogeneous styles, partial semantic alignment, temporal discontinuity, visual clutter, and multi-step inference.

The paper positions this as a gap left by prior datasets such as FigureQA, DVQA, PlotQA, ChartQA, ChartLlama, ChartInfo, and SciGraphQA, which mostly evaluate reasoning in isolated chart contexts [2508.07630]. Even multi-chart work such as MultiChartQA is described as relying on controlled, uniform layouts and not fully capturing real-world heterogeneity [2508.07630]. This places InterChart in contrast with earlier single-chart systems such as ChartNet, which formulates reasoning over bar charts and pie charts as a visual reasoning problem using MAC-Networks and supports open-ended chart-specific answers via localization plus OCR [1911.09375]. A plausible implication is that InterChart tests not just chart parsing, but whether parsing can be coordinated across multiple visual contexts with nontrivial semantic alignment.

The benchmark’s central claim is that real chart reasoning often depends on information that is distributed rather than localized. Its question families include entity inference, trend correlation, numerical estimation, and abstract multi-step reasoning, all of which require models to connect signals across separate charts instead of extracting a single local fact [2508.07630].

## 2. Benchmark design and dataset composition

InterChart contains **5,214 validated QA pairs** across **1,012 multi-chart contexts** and **2,706 unique chart images** [2508.07630]. Its design is explicitly tiered so that failure modes can be localized by complexity level rather than collapsed into one undifferentiated score.

The first tier, **DECAF**—Decomposed Elementary Charts with Answerable Facts—is the simplest subset. It is intended to establish baseline chart understanding in a reduced-complexity setting using single-variable charts, simplified visual layouts, and decomposed versions of compound figures [2508.07630]. The authors selected compound charts from ChartQA, ChartLlama, ChartInfo, and DVQA; chart types include vertical bar, horizontal bar, line, box plot, dot plot, and heat map [2508.07630]. When underlying tables were missing, **DePlot** was used to regenerate them from chart images, after which a custom decomposition script extracted rows, aligned them with legends and axis labels, and rendered simplified single-variable charts using **Plotly** [2508.07630]. This process produced **355 compound charts**, **1,188 decomposed charts**, and **2,809 QA pairs** [2508.07630].

The second tier, **SPECTRA**—Synthetic Plots for Event-based Correlated Trend Reasoning and Analysis—tests integration across synthetically aligned chart pairs that share a common axis but differ in style [2508.07630]. The authors created structured tables with shared axes, used **Gemini 1.5 Pro** to generate tabular data with controlled variability, then rendered visually diverse charts through a human-in-the-loop generation process that manually checked balanced axis scales, legend consistency, and chart type diversity [2508.07630]. This subset contains **1,717 QA pairs**, **333 visual context sets**, and **870 unique charts** [2508.07630].

The third tier, **STORM**—Sequential Temporal reasoning Over Real-world Multi-domain charts—is the hardest subset and evaluates semantic inference over visually complex real-world chart pairs [2508.07630]. Charts were crawled from **Our World in Data**, paired using metadata with aligned topics or axes, manually reviewed for coherence, and then assembled using the STORM pipeline described in the appendix [2508.07630]. STORM contains **768 QA pairs**, **324 original chart sets**, and **648 unique images** [2508.07630]. The paper attributes its difficulty to style and domain mismatch, partial semantic alignment, irregular axes, and the need for multi-step inference combining charts and metadata [2508.07630].

## 3. Reasoning tasks and validation methodology

InterChart covers four main reasoning styles: **entity inference**, **trend correlation**, **numerical estimation**, and **abstract multi-step reasoning** [2508.07630]. These categories are distributed across the three tiers rather than confined to a single subset, which allows the benchmark to probe how the same broad reasoning mode behaves under increasing visual and semantic complexity.

DECAF emphasizes factual lookup, comparison, and parallel reasoning over clearly presented values [2508.07630]. SPECTRA extends this to low-level reasoning such as totals and averages, trend analysis including directional inference and predictions, and scenario-based inference involving multi-condition comparisons [2508.07630]. The paper explicitly notes that correlated questions are harder than independent ones in this setting [2508.07630]. STORM focuses on **Range estimation**, **Abstract numerical reasoning**, and **Entity inference**, with the last of these still far from solved despite being easiest among the three [2508.07630].

The benchmark uses a multi-stage filtering and validation process. For DECAF and SPECTRA, candidate QA samples underwent **LLM-based acceptability checks** to remove ambiguous or malformed items and then human review by **6 graduate-level annotators** [2508.07630]. STORM was independently verified by **2 annotators**, with arbitration for disagreements [2508.07630]. The filtering statistics are substantial: DECAF and SPECTRA were reduced from **13,000** and **4,800** candidate QA samples to **2,809** and **1,717** final QA pairs, respectively [2508.07630]. For STORM, reported agreement statistics are **Cohen’s \(\kappa = 70.63\%\)** and **Jaccard index = 94.75\%**, which the paper presents as evidence of consistent annotation despite high task complexity [2508.07630].

This emphasis on validation differentiates InterChart from single-chart synthetic datasets built primarily to test structured reasoning on controlled layouts. For example, ChartNet creates a synthetic dataset of bar and pie charts with **10 question-answer pairs per chart image**, plus bounding-box annotations for chart text answers, and evaluates compositional reasoning over position, size, and order relations within a single chart [1911.09375]. InterChart instead prioritizes cross-chart distribution of evidence and multi-context validation [2508.07630].

## 4. Evaluation protocol and model setup

A central feature of InterChart is its rejection of exact string match as the primary evaluator for chart QA. The paper argues that such matching is brittle because answers may be paraphrased, approximate numerically, or expressed in equivalent units [2508.07630]. Instead, it uses **LLM-assisted semantic judging** with three evaluators—**Gemini 1.5 Flash**, **Phi-4**, and **Qwen2.5-7B-Instruct**—each of which receives the question, the ground-truth answer, and the model output, then returns a binary correctness decision and reasoning [2508.07630]. Final correctness is determined by **majority voting** across the judges [2508.07630].

On **10,000 sampled responses**, all three evaluators agreed in **78.67\%** of cases [2508.07630]. This is presented as a reliability check for semantic evaluation under conditions where lexical equivalence is inadequate. The choice parallels concerns raised in ChartDiff, where lexical metrics such as ROUGE reward overlap but do not reliably capture whether the system understood the comparative relationship correctly [2603.28902]. ChartDiff reports a strong **Pearson correlation of 0.91** between its GPT Score and human judgments on **300 sampled summaries**, reinforcing the general point that human-aligned chart evaluation often requires semantic rather than lexical scoring [2603.28902].

The model suite evaluated on InterChart includes both closed and open models: **Gemini 1.5 Pro**, **GPT-4o mini**, **Qwen2-VL-7B-Instruct**, **MiniCPM-V-2.6**, **InternVL-2-8B**, and **Idefics3-8B-LLaMA3**, as well as structured baselines including **DePlot**, **Chart-to-Text / C2T**, and **DePlot++** [2508.07630]. The experiments test **Zero-shot**, **Zero-shot CoT**, and **Few-shot CoT with directives**, compare **Combined** versus **Interleaved** visual input formats, and include a two-stage structured pipeline comprising chart-to-table extraction followed by table-based QA [2508.07630].

This evaluation design allows InterChart to separate several often conflated questions: whether raw visual inputs or structured intermediates are better, whether prompting helps, and whether alternative chart presentation formats mitigate reasoning failures [2508.07630].

## 5. Empirical findings

The primary empirical result is a consistent decline in accuracy from DECAF to SPECTRA to STORM across model families [2508.07630]. Under the main setting—**Combined visual context, zero-shot**—the paper reports the following trajectories:

| Model | DECAF | SPECTRA | STORM |
|---|---:|---:|---:|
| Gemini 1.5 Pro | 65.2 | 59.1 | 34.8 |
| GPT-4o mini | 59.3 | 45.6 | 29.7 |
| Qwen2-VL | 50.2 | 32.8 | 28.9 |
| MiniCPM-V-2.6 | 52.2 | 32.4 | 21.5 |
| InternVL-2-8B | 40.0 | 26.6 | 24.8 |
| Idefics3-8B-LLaMA3 | 39.3 | 19.4 | 11.1 |

These results support the paper’s claim that models handle simplified decomposed charts best, struggle more on synthetic multi-chart reasoning, and perform worst on real-world multi-domain chart pairs [2508.07630].

A second major result is that **decomposition helps** [2508.07630]. The paper reports that chart-to-table and decomposition-based methods often outperform raw image reasoning on simpler subsets, which it interprets as evidence that models benefit from reduced visual clutter, better-aligned titles and metadata, and conversion of dense figures into structured units [2508.07630]. For example, **Gemini 1.5 Pro** attains **69.9\%** on chart-to-table DECAF, and **DePlot++** improves over **DePlot** from **54.3\%** to **63.2\%** on DECAF and from **57.9\%** to **58.1\%** on SPECTRA [2508.07630].

However, structured tables are not universally advantageous. On STORM, **Gemini 1.5 Pro** drops from **34.8\%** with visual inputs to **29.5\%** with tables, while **C2T** performs especially poorly at **14.7\%** [2508.07630]. The paper interprets this as evidence that the hard part in real-world multi-chart reasoning is not merely value extraction; it also involves semantic alignment, temporal context, and preservation of cross-chart structure [2508.07630]. This observation is consistent with work such as ChartReader, which argues that chart comprehension should not reduce chart understanding to text-only reasoning over OCR output or ground-truth tables, because visual structure and semantics matter materially to performance [2304.02173].

Prompting yields limited gains. Chain-of-thought helps in DECAF and SPECTRA but has limited effect on STORM, suggesting that verbalized reasoning cannot compensate for deep visual-semantic complexity [2508.07630]. Likewise, the choice between **Interleaved** and **Combined** input formats is not a universal solution: it sometimes helps and sometimes hurts, but does not resolve the core cross-chart reasoning problem [2508.07630].

## 6. Relation to adjacent chart-understanding research

InterChart sits at the intersection of chart QA, chart derendering, chart summarization, and multi-chart comparative reasoning. Its most direct conceptual neighbor is **ChartDiff**, which introduces the first large-scale benchmark for **cross-chart comparative summarization** and contains **8,541 chart pairs** with summaries describing differences in trends, fluctuations, and anomalies [2603.28902]. ChartDiff evaluates systems on a generative comparison task rather than question answering, but the two benchmarks share the premise that chart understanding must move beyond isolated chart interpretation.

The distinction between them is methodological. InterChart is diagnostic and QA-oriented, explicitly structured into DECAF, SPECTRA, and STORM to measure how models cope with decomposed, synthetic aligned, and real-world complex chart settings [2508.07630]. ChartDiff instead asks models to generate a concise comparison paragraph and emphasizes the mismatch between ROUGE and human-aligned evaluation, reporting that frontier general-purpose models have the best GPT-based quality while specialized and pipeline methods often achieve higher ROUGE but lower human-aligned quality [2603.28902]. This suggests that InterChart and ChartDiff probe related but non-identical competencies: the former emphasizes distributed evidence integration for QA, while the latter emphasizes comparative summarization fidelity.

InterChart also differs from single-chart reasoning systems such as **ChartNet**, which adapts **MAC-Networks** to bar and pie chart QA and introduces a regression head that predicts a bounding box around answer text, later read by OCR [1911.09375]. ChartNet shows that compositional reasoning is effective for chart QA within a single image, achieving **91.42\%** accuracy on pie chart reasoning, **98.14\%** on bar chart reasoning, and **0.84** mean IoU for open-ended answer localization [1911.09375]. InterChart generalizes the challenge from within-chart compositionality to cross-chart distribution of facts, where even strong models degrade sharply [2508.07630].

A further point of contact is with chart-structure extraction. **ChartReader** integrates chart derendering and comprehension in one framework, using a transformer-based chart component detector and an extended pre-trained vision-language model for chart-to-X tasks [2304.02173]. **CHARTER** provides a four-stage pipeline for detecting chart regions, extracting graphical elements and OCR, and reconstructing source tables across bar, pie, line, and scatter plots using heatmaps and synthetic-data training [2111.14103]. **ChartT5** argues that chart understanding improves when models learn to recover the hidden table behind a chart image via cross-modal pre-training on plot-table pairs [2305.18641]. InterChart’s findings that structured representations help on easier subsets but can hurt on STORM indicate that these extraction-centric paradigms are necessary but not sufficient for robust multi-chart reasoning [2508.07630].

Finally, **ChartInstruct** broadens chart understanding through instruction tuning on **191K instructions** generated with **71K charts**, supporting tasks such as summarization, open-ended QA, fact checking, chain-of-thought reasoning, code generation, anomaly detection, and forecasting [2403.09028]. A plausible implication is that instruction tuning may improve flexibility across task formulations, but InterChart indicates that flexibility alone does not remove the difficulty of semantically aligning distributed information across several visuals [2508.07630].

## 7. Significance, limitations, and future directions

InterChart’s significance lies in reframing chart understanding as a problem of **distributed reasoning** rather than isolated figure reading [2508.07630]. Its results show that current multimodal systems are much more capable when evidence is simplified, decomposed, or localized than when they must compare multiple charts, align different semantics, integrate metadata with visual cues, reason across time or domains, or perform multi-step numerical synthesis [2508.07630]. The benchmark therefore functions not only as an evaluation set but also as a diagnostic instrument for identifying where contemporary VLMs fail.

The benchmark also makes a methodological contribution through its three-tier design. DECAF isolates factual chart reading under reduced clutter; SPECTRA introduces controlled cross-chart integration; and STORM exposes models to real-world semantic and stylistic heterogeneity [2508.07630]. This tiering helps distinguish whether a model’s weakness arises from basic chart parsing, correlated reasoning, or true semantic integration across complex visuals.

At the same time, the paper does not claim that decomposition or structured parsing fully solves the problem. The reported gains from decomposition on simpler subsets, combined with deterioration from table-based methods in STORM, indicate that both visual and structured representations are incomplete on their own [2508.07630]. This suggests that future systems may need hybrid representations that preserve chart geometry, metadata, temporal structure, and entity alignment simultaneously. The paper itself suggests extending InterChart to **infographics**, **annotated scientific plots**, **hybrid layouts**, and **multilingual question sets**, and proposes research directions including **neuro-symbolic approaches**, **retrieval-augmented methods**, **layout-aware fine-tuning**, **grounded chain-of-thought prompting**, **structured parsing outputs**, and **multimodal summarization agents for multi-chart analytics** [2508.07630].

A broader implication, supported by adjacent work, is that progress in chart intelligence is fragmenting into several complementary fronts: single-chart compositional reasoning [1911.09375], unified chart derendering and comprehension [2304.02173], multi-type extraction from document charts [2111.14103], chart-table pre-training [2305.18641], instruction-tuned chart assistants [2403.09028], cross-chart summarization [2603.28902], and, with InterChart, rigorous evaluation of multi-chart distributed reasoning [2508.07630]. InterChart’s role within this landscape is to make explicit that multi-chart understanding is not a minor extension of single-chart QA, but a distinct and substantially harder problem.

Source: https://www.emergentmind.com/topics/interchart