InterChart: Multi-Chart Visual Reasoning Benchmark
- InterChart is a diagnostic benchmark for multi-chart visual reasoning that tests models’ ability to integrate distributed information from related charts.
- It organizes challenges into three tiers—DECAF, SPECTRA, and STORM—to assess reasoning from basic factual lookup to complex semantic integration.
- Empirical results show that while decomposing charts boosts performance on simpler datasets, models struggle with accuracy on real-world, heterogeneous chart tasks.
Searching arXiv for the InterChart paper and closely related chart-understanding work to ground the article in current literature. InterChart is a diagnostic benchmark for multi-chart visual reasoning that evaluates whether vision-LLMs can answer questions by connecting information distributed across $2$–$3$ related charts rather than reading a single chart in isolation. It is designed for settings such as scientific reporting, financial analysis, and public policy dashboards, where answers often depend on comparison, alignment, or synthesis across multiple visuals rather than extraction from one figure alone (Iyengar et al., 11 Aug 2025). The benchmark organizes this problem into three tiers of increasing difficulty—DECAF, SPECTRA, and STORM—and reports that state-of-the-art open and closed-source models exhibit steep accuracy declines as chart complexity increases, while also showing that decomposing multi-entity charts into simpler visual units can improve performance (Iyengar et al., 11 Aug 2025). In the broader chart-understanding literature, InterChart occupies the multi-chart end of a spectrum that includes single-chart question answering systems such as ChartNet (Sharma et al., 2019), unified chart derendering and comprehension frameworks such as ChartReader (Cheng et al., 2023), and comparative chart summarization benchmarks such as ChartDiff (Ye, 30 Mar 2026).
1. Concept and motivation
InterChart targets a specific failure mode in contemporary chart-understanding systems: they are often competent at reasoning over isolated, visually uniform charts, but much weaker when relevant evidence is decomposed across several related figures (Iyengar et al., 11 Aug 2025). The benchmark is motivated by the observation that many practical chart-analysis tasks require cross-chart integration under conditions of heterogeneous styles, partial semantic alignment, temporal discontinuity, visual clutter, and multi-step inference.
The paper positions this as a gap left by prior datasets such as FigureQA, DVQA, PlotQA, ChartQA, ChartLlama, ChartInfo, and SciGraphQA, which mostly evaluate reasoning in isolated chart contexts (Iyengar et al., 11 Aug 2025). Even multi-chart work such as MultiChartQA is described as relying on controlled, uniform layouts and not fully capturing real-world heterogeneity (Iyengar et al., 11 Aug 2025). This places InterChart in contrast with earlier single-chart systems such as ChartNet, which formulates reasoning over bar charts and pie charts as a visual reasoning problem using MAC-Networks and supports open-ended chart-specific answers via localization plus OCR (Sharma et al., 2019). A plausible implication is that InterChart tests not just chart parsing, but whether parsing can be coordinated across multiple visual contexts with nontrivial semantic alignment.
The benchmark’s central claim is that real chart reasoning often depends on information that is distributed rather than localized. Its question families include entity inference, trend correlation, numerical estimation, and abstract multi-step reasoning, all of which require models to connect signals across separate charts instead of extracting a single local fact (Iyengar et al., 11 Aug 2025).
2. Benchmark design and dataset composition
InterChart contains 5,214 validated QA pairs across 1,012 multi-chart contexts and 2,706 unique chart images (Iyengar et al., 11 Aug 2025). Its design is explicitly tiered so that failure modes can be localized by complexity level rather than collapsed into one undifferentiated score.
The first tier, DECAF—Decomposed Elementary Charts with Answerable Facts—is the simplest subset. It is intended to establish baseline chart understanding in a reduced-complexity setting using single-variable charts, simplified visual layouts, and decomposed versions of compound figures (Iyengar et al., 11 Aug 2025). The authors selected compound charts from ChartQA, ChartLlama, ChartInfo, and DVQA; chart types include vertical bar, horizontal bar, line, box plot, dot plot, and heat map (Iyengar et al., 11 Aug 2025). When underlying tables were missing, DePlot was used to regenerate them from chart images, after which a custom decomposition script extracted rows, aligned them with legends and axis labels, and rendered simplified single-variable charts using Plotly (Iyengar et al., 11 Aug 2025). This process produced 355 compound charts, 1,188 decomposed charts, and 2,809 QA pairs (Iyengar et al., 11 Aug 2025).
The second tier, SPECTRA—Synthetic Plots for Event-based Correlated Trend Reasoning and Analysis—tests integration across synthetically aligned chart pairs that share a common axis but differ in style (Iyengar et al., 11 Aug 2025). The authors created structured tables with shared axes, used Gemini 1.5 Pro to generate tabular data with controlled variability, then rendered visually diverse charts through a human-in-the-loop generation process that manually checked balanced axis scales, legend consistency, and chart type diversity (Iyengar et al., 11 Aug 2025). This subset contains 1,717 QA pairs, 333 visual context sets, and 870 unique charts (Iyengar et al., 11 Aug 2025).
The third tier, STORM—Sequential Temporal reasoning Over Real-world Multi-domain charts—is the hardest subset and evaluates semantic inference over visually complex real-world chart pairs (Iyengar et al., 11 Aug 2025). Charts were crawled from Our World in Data, paired using metadata with aligned topics or axes, manually reviewed for coherence, and then assembled using the STORM pipeline described in the appendix (Iyengar et al., 11 Aug 2025). STORM contains 768 QA pairs, 324 original chart sets, and 648 unique images (Iyengar et al., 11 Aug 2025). The paper attributes its difficulty to style and domain mismatch, partial semantic alignment, irregular axes, and the need for multi-step inference combining charts and metadata (Iyengar et al., 11 Aug 2025).
3. Reasoning tasks and validation methodology
InterChart covers four main reasoning styles: entity inference, trend correlation, numerical estimation, and abstract multi-step reasoning (Iyengar et al., 11 Aug 2025). These categories are distributed across the three tiers rather than confined to a single subset, which allows the benchmark to probe how the same broad reasoning mode behaves under increasing visual and semantic complexity.
DECAF emphasizes factual lookup, comparison, and parallel reasoning over clearly presented values (Iyengar et al., 11 Aug 2025). SPECTRA extends this to low-level reasoning such as totals and averages, trend analysis including directional inference and predictions, and scenario-based inference involving multi-condition comparisons (Iyengar et al., 11 Aug 2025). The paper explicitly notes that correlated questions are harder than independent ones in this setting (Iyengar et al., 11 Aug 2025). STORM focuses on Range estimation, Abstract numerical reasoning, and Entity inference, with the last of these still far from solved despite being easiest among the three (Iyengar et al., 11 Aug 2025).
The benchmark uses a multi-stage filtering and validation process. For DECAF and SPECTRA, candidate QA samples underwent LLM-based acceptability checks to remove ambiguous or malformed items and then human review by 6 graduate-level annotators (Iyengar et al., 11 Aug 2025). STORM was independently verified by 2 annotators, with arbitration for disagreements (Iyengar et al., 11 Aug 2025). The filtering statistics are substantial: DECAF and SPECTRA were reduced from 13,000 and 4,800 candidate QA samples to 2,809 and 1,717 final QA pairs, respectively (Iyengar et al., 11 Aug 2025). For STORM, reported agreement statistics are Cohen’s and Jaccard index = 94.75\%, which the paper presents as evidence of consistent annotation despite high task complexity (Iyengar et al., 11 Aug 2025).
This emphasis on validation differentiates InterChart from single-chart synthetic datasets built primarily to test structured reasoning on controlled layouts. For example, ChartNet creates a synthetic dataset of bar and pie charts with 10 question-answer pairs per chart image, plus bounding-box annotations for chart text answers, and evaluates compositional reasoning over position, size, and order relations within a single chart (Sharma et al., 2019). InterChart instead prioritizes cross-chart distribution of evidence and multi-context validation (Iyengar et al., 11 Aug 2025).
4. Evaluation protocol and model setup
A central feature of InterChart is its rejection of exact string match as the primary evaluator for chart QA. The paper argues that such matching is brittle because answers may be paraphrased, approximate numerically, or expressed in equivalent units (Iyengar et al., 11 Aug 2025). Instead, it uses LLM-assisted semantic judging with three evaluators—Gemini 1.5 Flash, Phi-4, and Qwen2.5-7B-Instruct—each of which receives the question, the ground-truth answer, and the model output, then returns a binary correctness decision and reasoning (Iyengar et al., 11 Aug 2025). Final correctness is determined by majority voting across the judges (Iyengar et al., 11 Aug 2025).
On 10,000 sampled responses, all three evaluators agreed in 78.67\% of cases (Iyengar et al., 11 Aug 2025). This is presented as a reliability check for semantic evaluation under conditions where lexical equivalence is inadequate. The choice parallels concerns raised in ChartDiff, where lexical metrics such as ROUGE reward overlap but do not reliably capture whether the system understood the comparative relationship correctly (Ye, 30 Mar 2026). ChartDiff reports a strong Pearson correlation of 0.91 between its GPT Score and human judgments on 300 sampled summaries, reinforcing the general point that human-aligned chart evaluation often requires semantic rather than lexical scoring (Ye, 30 Mar 2026).
The model suite evaluated on InterChart includes both closed and open models: Gemini 1.5 Pro, GPT-4o mini, Qwen2-VL-7B-Instruct, MiniCPM-V-2.6, InternVL-2-8B, and Idefics3-8B-LLaMA3, as well as structured baselines including DePlot, Chart-to-Text / C2T, and DePlot++ (Iyengar et al., 11 Aug 2025). The experiments test Zero-shot, Zero-shot CoT, and Few-shot CoT with directives, compare Combined versus Interleaved visual input formats, and include a two-stage structured pipeline comprising chart-to-table extraction followed by table-based QA (Iyengar et al., 11 Aug 2025).
This evaluation design allows InterChart to separate several often conflated questions: whether raw visual inputs or structured intermediates are better, whether prompting helps, and whether alternative chart presentation formats mitigate reasoning failures (Iyengar et al., 11 Aug 2025).
5. Empirical findings
The primary empirical result is a consistent decline in accuracy from DECAF to SPECTRA to STORM across model families (Iyengar et al., 11 Aug 2025). Under the main setting—Combined visual context, zero-shot—the paper reports the following trajectories:
| Model | DECAF | SPECTRA | STORM |
|---|---|---|---|
| Gemini 1.5 Pro | 65.2 | 59.1 | 34.8 |
| GPT-4o mini | 59.3 | 45.6 | 29.7 |
| Qwen2-VL | 50.2 | 32.8 | 28.9 |
| MiniCPM-V-2.6 | 52.2 | 32.4 | 21.5 |
| InternVL-2-8B | 40.0 | 26.6 | 24.8 |
| Idefics3-8B-LLaMA3 | 39.3 | 19.4 | 11.1 |
These results support the paper’s claim that models handle simplified decomposed charts best, struggle more on synthetic multi-chart reasoning, and perform worst on real-world multi-domain chart pairs (Iyengar et al., 11 Aug 2025).
A second major result is that decomposition helps (Iyengar et al., 11 Aug 2025). The paper reports that chart-to-table and decomposition-based methods often outperform raw image reasoning on simpler subsets, which it interprets as evidence that models benefit from reduced visual clutter, better-aligned titles and metadata, and conversion of dense figures into structured units (Iyengar et al., 11 Aug 2025). For example, Gemini 1.5 Pro attains 69.9\% on chart-to-table DECAF, and DePlot++ improves over DePlot from 54.3\% to 63.2\% on DECAF and from 57.9\% to 58.1\% on SPECTRA (Iyengar et al., 11 Aug 2025).
However, structured tables are not universally advantageous. On STORM, Gemini 1.5 Pro drops from 34.8\% with visual inputs to 29.5\% with tables, while C2T performs especially poorly at 14.7\% (Iyengar et al., 11 Aug 2025). The paper interprets this as evidence that the hard part in real-world multi-chart reasoning is not merely value extraction; it also involves semantic alignment, temporal context, and preservation of cross-chart structure (Iyengar et al., 11 Aug 2025). This observation is consistent with work such as ChartReader, which argues that chart comprehension should not reduce chart understanding to text-only reasoning over OCR output or ground-truth tables, because visual structure and semantics matter materially to performance (Cheng et al., 2023).
Prompting yields limited gains. Chain-of-thought helps in DECAF and SPECTRA but has limited effect on STORM, suggesting that verbalized reasoning cannot compensate for deep visual-semantic complexity (Iyengar et al., 11 Aug 2025). Likewise, the choice between Interleaved and Combined input formats is not a universal solution: it sometimes helps and sometimes hurts, but does not resolve the core cross-chart reasoning problem (Iyengar et al., 11 Aug 2025).
6. Relation to adjacent chart-understanding research
InterChart sits at the intersection of chart QA, chart derendering, chart summarization, and multi-chart comparative reasoning. Its most direct conceptual neighbor is ChartDiff, which introduces the first large-scale benchmark for cross-chart comparative summarization and contains 8,541 chart pairs with summaries describing differences in trends, fluctuations, and anomalies (Ye, 30 Mar 2026). ChartDiff evaluates systems on a generative comparison task rather than question answering, but the two benchmarks share the premise that chart understanding must move beyond isolated chart interpretation.
The distinction between them is methodological. InterChart is diagnostic and QA-oriented, explicitly structured into DECAF, SPECTRA, and STORM to measure how models cope with decomposed, synthetic aligned, and real-world complex chart settings (Iyengar et al., 11 Aug 2025). ChartDiff instead asks models to generate a concise comparison paragraph and emphasizes the mismatch between ROUGE and human-aligned evaluation, reporting that frontier general-purpose models have the best GPT-based quality while specialized and pipeline methods often achieve higher ROUGE but lower human-aligned quality (Ye, 30 Mar 2026). This suggests that InterChart and ChartDiff probe related but non-identical competencies: the former emphasizes distributed evidence integration for QA, while the latter emphasizes comparative summarization fidelity.
InterChart also differs from single-chart reasoning systems such as ChartNet, which adapts MAC-Networks to bar and pie chart QA and introduces a regression head that predicts a bounding box around answer text, later read by OCR (Sharma et al., 2019). ChartNet shows that compositional reasoning is effective for chart QA within a single image, achieving 91.42\% accuracy on pie chart reasoning, 98.14\% on bar chart reasoning, and 0.84 mean IoU for open-ended answer localization (Sharma et al., 2019). InterChart generalizes the challenge from within-chart compositionality to cross-chart distribution of facts, where even strong models degrade sharply (Iyengar et al., 11 Aug 2025).
A further point of contact is with chart-structure extraction. ChartReader integrates chart derendering and comprehension in one framework, using a transformer-based chart component detector and an extended pre-trained vision-LLM for chart-to-X tasks (Cheng et al., 2023). CHARTER provides a four-stage pipeline for detecting chart regions, extracting graphical elements and OCR, and reconstructing source tables across bar, pie, line, and scatter plots using heatmaps and synthetic-data training (Shtok et al., 2021). ChartT5 argues that chart understanding improves when models learn to recover the hidden table behind a chart image via cross-modal pre-training on plot-table pairs (Zhou et al., 2023). InterChart’s findings that structured representations help on easier subsets but can hurt on STORM indicate that these extraction-centric paradigms are necessary but not sufficient for robust multi-chart reasoning (Iyengar et al., 11 Aug 2025).
Finally, ChartInstruct broadens chart understanding through instruction tuning on 191K instructions generated with 71K charts, supporting tasks such as summarization, open-ended QA, fact checking, chain-of-thought reasoning, code generation, anomaly detection, and forecasting (Masry et al., 2024). A plausible implication is that instruction tuning may improve flexibility across task formulations, but InterChart indicates that flexibility alone does not remove the difficulty of semantically aligning distributed information across several visuals (Iyengar et al., 11 Aug 2025).
7. Significance, limitations, and future directions
InterChart’s significance lies in reframing chart understanding as a problem of distributed reasoning rather than isolated figure reading (Iyengar et al., 11 Aug 2025). Its results show that current multimodal systems are much more capable when evidence is simplified, decomposed, or localized than when they must compare multiple charts, align different semantics, integrate metadata with visual cues, reason across time or domains, or perform multi-step numerical synthesis (Iyengar et al., 11 Aug 2025). The benchmark therefore functions not only as an evaluation set but also as a diagnostic instrument for identifying where contemporary VLMs fail.
The benchmark also makes a methodological contribution through its three-tier design. DECAF isolates factual chart reading under reduced clutter; SPECTRA introduces controlled cross-chart integration; and STORM exposes models to real-world semantic and stylistic heterogeneity (Iyengar et al., 11 Aug 2025). This tiering helps distinguish whether a model’s weakness arises from basic chart parsing, correlated reasoning, or true semantic integration across complex visuals.
At the same time, the paper does not claim that decomposition or structured parsing fully solves the problem. The reported gains from decomposition on simpler subsets, combined with deterioration from table-based methods in STORM, indicate that both visual and structured representations are incomplete on their own (Iyengar et al., 11 Aug 2025). This suggests that future systems may need hybrid representations that preserve chart geometry, metadata, temporal structure, and entity alignment simultaneously. The paper itself suggests extending InterChart to infographics, annotated scientific plots, hybrid layouts, and multilingual question sets, and proposes research directions including neuro-symbolic approaches, retrieval-augmented methods, layout-aware fine-tuning, grounded chain-of-thought prompting, structured parsing outputs, and multimodal summarization agents for multi-chart analytics (Iyengar et al., 11 Aug 2025).
A broader implication, supported by adjacent work, is that progress in chart intelligence is fragmenting into several complementary fronts: single-chart compositional reasoning (Sharma et al., 2019), unified chart derendering and comprehension (Cheng et al., 2023), multi-type extraction from document charts (Shtok et al., 2021), chart-table pre-training (Zhou et al., 2023), instruction-tuned chart assistants (Masry et al., 2024), cross-chart summarization (Ye, 30 Mar 2026), and, with InterChart, rigorous evaluation of multi-chart distributed reasoning (Iyengar et al., 11 Aug 2025). InterChart’s role within this landscape is to make explicit that multi-chart understanding is not a minor extension of single-chart QA, but a distinct and substantially harder problem.