ChartDQA: Comprehensive Chart QA Benchmark

• ChartDQA is a benchmark for chart question answering that maps chart representations and natural language queries to accurate answers.
• It features a synthetic dataset of 5,480 images covering 20 chart types, generated through an LLM-driven JSON pipeline and human filtering.
• The evaluation framework tests models on multiple reasoning levels—from factual extraction to higher-order inference—with metrics like Exact Match and JSON-alignment.

Searching arXiv for the named benchmark and closely related chart question answering work. {"query":"ChartDQA chart question answering benchmark 2025 ChartDQA arXiv", "max_results": 10} Retrieved arXiv results for "ChartDQA chart question answering benchmark 2025 ChartDQA arXiv". {"query":"Chart Question Answering from Real-World Analytical Narratives", "max_results": 5} ChartDQA denotes two closely related notions in the literature. In survey usage, it refers generically to chart question answering: learning a mapping from a chart representation and a natural-language question to an answer. In a benchmark-specific sense, it names a 2025 evaluation suite introduced with ChartScope for comprehensive chart understanding, designed to assess not only surface-level question answering but also recovery of the chart’s underlying data and reasoning over that data across a wide variety of chart types (Hoque et al., 2022, Fan et al., 18 Jul 2025). The benchmark-specific usage is the more precise contemporary meaning, but the broader task-level meaning remains common in reviews and adjacent work.

1. Definition and nomenclature

In the generic formulation, ChartDQA is the problem of learning a function

$$f:\bigl(C,Q\bigr)\longmapsto A \quad\text{where }C\in\mathcal{C},\;Q\in\mathcal{Q},\;A\in\mathcal{A}.$$

Here, $C$ is a chart representation, such as a chart image or extracted data table; $Q$ is a natural-language question; and $A$ is an admissible answer, such as a numeric value, label, or yes/no response (Hoque et al., 2022).

In the benchmark introduced by ChartScope, ChartDQA has a narrower and more demanding goal: to evaluate multimodal models not only on surface-level chart QA but also on their ability to recover and reason over the underlying data for a wide variety of chart types. Its design therefore combines question answering with explicit underlying-data alignment tasks, and it treats chart comprehension as a data-grounding problem rather than only an image-to-answer mapping (Fan et al., 18 Jul 2025).

A common source of confusion is the proximity of the names ChartDQA and ChartQA. ChartQA is an earlier benchmark centered on real-world bar, line, and pie charts with human-written and machine-generated questions, whereas ChartDQA emphasizes breadth across 20 chart types, multi-level understanding, and gold underlying data for every chart (Masry et al., 2022, Fan et al., 18 Jul 2025).

2. Dataset synthesis and composition

ChartDQA contains 5,480 images, each a unique chart instance, spanning 20 distinct types, including bar charts, line charts, pie charts, scatter plots, area charts, histograms, box plots, bubble charts, radar/spider plots, and stacked bar/area charts. All ChartDQA images are synthetic, generated via an LLM-driven JSON-data pipeline and Python-rendered charts, and then human-filtered for answerability and correctness (Fan et al., 18 Jul 2025).

The data-generation methodology is explicitly programmatic. First, 20 chart-type JSON templates are defined, including title, axes metadata, and data arrays. Second, a “data expert” LLM, such as GPT-4, is prompted to produce $M = 1000$ diversified JSON data files per chart type over varied topics such as energy markets, demographics, and finance. Third, a “code expert” LLM is prompted to generate $N = 400$ Python scripts per chart type, each rendering the data to an image with stylistic variation in color, grid, legends, and fonts. Fourth, scripts and data are composed to obtain roughly 5 million chart images, after which execution/OCR correctness filtering is applied and a subset of 5.48 k high-quality images is sampled for ChartDQA (Fan et al., 18 Jul 2025).

The benchmark is partitioned into train, validation, and test splits. The test split, approximately 1 k images, is human-verified on answerability and answer correctness; the train and validation splits comprise the remaining approximately 4.48 k images. Each sample is stored as a JSON record with fields {"chart_image_path":…, "chart_type":…, "raw_data":…, "question":…, "answer":…, "level":1|2|3}, and outputs are either a string or a JSON object, depending on the task variant (Fan et al., 18 Jul 2025).

The synthetic nature of ChartDQA is deliberate rather than incidental. The benchmark’s stated motivation is to widen chart-type coverage, diversify visual style, and make underlying data available as gold supervision for direct evaluation of chart-data alignment (Fan et al., 18 Jul 2025).

3. Understanding levels and task formalization

ChartDQA organizes chart understanding into three question-answering levels plus a data-alignment setting. Level 1 corresponds to factual extraction or literal QA; Level 2 to inferential QA; Level 3 to insight-generation or reasoning QA; and the underlying-data alignment tasks require extraction of the chart’s underlying data in JSON or tabular form as an intermediate or auxiliary output (Fan et al., 18 Jul 2025).

Level/task	Definition
Level 1	retrieving explicit values or labels shown in the chart
Level 2	simple computations or comparisons over chart data
Level 3	multi-step or higher-order reasoning, possibly requiring arithmetic chains on the raw data
Underlying-data alignment	extraction of the chart’s underlying data as JSON or tabular output

The formalization introduces a chart image $C$ and an underlying data matrix $D \in \mathbb{R}^{k\times 2}$, where $k$ is the number of data points and the two columns represent $x$ and $C$0 values. Literal questions are modeled as $C$1, with answers directly read from either $C$2 or axis labels; the given template is

$C$3

Inferential questions require a simple function over $C$4, for example

$C$5

Reasoning questions require composite computations on $C$6, such as

$C$7

The data-driven QA setting is defined as a two-turn exchange: first output the chart data as JSON, then answer a Level 3 question using the extracted JSON. A further variant, JSON-only QA, replaces the image with $C$8 plus README text while preserving the reasoning templates from Levels 1–3 (Fan et al., 18 Jul 2025).

The benchmark’s sample instance is a bar chart of monthly sales from January to June with underlying data $C$9. On this example, the Level 1 answer to “What were sales in March?” is “12”; the Level 2 answer to “Which month had the highest sales?” is “June”; the Level 3 answer to “What is the percentage increase from April to May?” is “$Q$0”; and the data-alignment output is the corresponding JSON object with x and y arrays (Fan et al., 18 Jul 2025).

4. Evaluation protocol and metrics

ChartDQA uses Exact Match (EM), F1, and JSON-alignment EM. The EM metric is defined as

$Q$1

where $Q$2 is the model output and $Q$3 is the gold answer. For multi-token or numeric extraction, the benchmark also reports F1 via the standard precision, recall, and harmonic-mean definitions. JSON-alignment EM treats the predicted JSON $Q$4 as a serialized string and computes exact match against the gold serialized JSON. For numeric reasoning questions, a numeric tolerance band of $Q$5–$Q$6 relative may be applied before an answer is counted as correct (Fan et al., 18 Jul 2025).

Reporting is stratified by chart-type family. ChartDQA reports separate EM on Basic types—bar, line, and pie—and Advanced types—the other 17 chart categories. Aggregate EM is the average over all types. This separation is intended to distinguish performance on canonical chart families from performance on broader visual diversity (Fan et al., 18 Jul 2025).

The evaluation toolkit includes data loaders for ChartDQA, EM/F1 calculators, JSON-parsing validators, and sample inference notebooks for LVLMs. Code and evaluation scripts are published alongside the benchmark (Fan et al., 18 Jul 2025).

Metric design is central to the benchmark’s scope. Because underlying-data extraction is itself evaluated, ChartDQA can distinguish correct final answers that arise from correct data grounding from those that may be produced by shortcut exploitation or memorized visual heuristics. That distinction is a defining departure from benchmarks that score only answer correctness (Fan et al., 18 Jul 2025).

5. Relation to prior benchmarks and task variants

The survey literature places chart question answering within a sequence of datasets that differ by chart types, scale, question-generation method, and answer vocabulary. FigureQA contains 180 K charts and 2.3 M question-answer pairs; DVQA contains 300 K charts and 3.4 M question-answer pairs; LeafQA contains 240 K charts and 2.0 M question-answer pairs; PlotQA contains 224 K charts and 28 M question-answer pairs; and ChartQA contains 4.8 K charts and 9.6 K question-answer pairs over bar, line, and pie charts (Hoque et al., 2022).

ChartQA is especially relevant as the immediate predecessor in mainstream chart reasoning benchmarks. It was built from 20,882 real-world charts crawled from Statista, Pew, Our World In Data, and OECD, and includes 9,608 human-written QA pairs plus 23,111 machine-generated QA pairs. Its human questions were designed to emphasize compositional reasoning and visual referencing, and a sample of 300 human questions showed approximately 24% data retrieval only, approximately 7% visual-only, 43% compositional, and 33% visual + compositional (Masry et al., 2022).

ChartDQA differentiates itself along four axes stated explicitly in its description. Its breadth is 20 chart types, versus 3–18 in prior work; its depth lies in multi-level QA sets and explicit data alignment tasks; it provides gold raw data for every chart; and its style variation is intended to discourage reliance on fixed visual templates or OCR-only shortcuts (Fan et al., 18 Jul 2025).

The task family has also expanded in two orthogonal directions. One is document-level chart question answering, where the system first performs document layout analysis to detect chart regions and then answers questions over the extracted chart crop; the DCQA benchmark contains 50,010 synthetic documents and 699,051 questions over 6 chart styles and 30 subtypes (Wu et al., 2023). The other is ecologically grounded chart QA from analytical narratives: Hutchinson et al. construct a dataset from “literate visualization” notebooks, yielding 103 distinct visualization images and 205 multiple-choice question-answer pairs, of which 75 refer to more than one image or view and 33 are labeled “Cannot be determined from the visualization(s)” (Hutchinson et al., 2 Jul 2025).

A plausible implication is that contemporary chart-QA evaluation is bifurcating into complementary regimes. ChartDQA emphasizes synthetic breadth, stylistic variation, and explicit chart-data alignment, whereas notebook-derived narrative benchmarks emphasize ecologically valid reasoning workflows, multi-view charts, and natural-language grounding in real analytical narratives (Fan et al., 18 Jul 2025, Hutchinson et al., 2 Jul 2025).

6. Representative model families

The benchmark emerged within a broader methodological landscape that includes explicit compositional reasoning, question-guided visual grounding, and chain-of-thought supervision with reinforcement learning.

GoT-CQA models a chart question as a directed acyclic Graph-of-Thought composed of operator nodes of types Loc, Num, and Log, then executes a matching auto-compositional reasoning framework over chart features. On adjacent benchmarks, it achieved 67.5% on ChartQA, 92.8% overall on PlotQA-D1, and 78.3% overall on PlotQA-D2; ablations reported an approximately 1.2–1.3% drop without explicit GoT guidance and found that Loc+Num+Log outperformed coarser operator sets (Zhang et al., 2024).

ChartFormer and QDChart approach chart QA through robust chart component recognition and question-guided deformable co-attention. ChartFormer is an end-to-end instance-segmentation model for bars, lines, pies, titles, legends, and axes, while QDChart fuses ChartFormer visual features, complementary Swin-Transformer features, and question embeddings from mBART. On ExcelChart400K, ChartFormer improved mAP from 61.7 to 64.9 over Mask2Former, and on ChartQA, QDChart reached 57.2% Relaxed Accuracy versus 56.0% for Pix2Struct (Zheng et al., 2024).

Chart-R1 targets complex chart reasoning with two-stage training: Chart-COT, using step-by-step chain-of-thought supervision, and Chart-RFT, using numerically sensitive reinforcement fine-tuning with GRPO. Built on Qwen2.5-VL-7B-Instruct, it was trained with 228 K ChartRQA-SFT examples and 30 K ChartRQA-RL examples, and it reported 91.0 on ChartQA, 44.0 on ChartQAPro, and 52.1 / 49.9 on ChartRQA single/multi (Chen et al., 21 Jul 2025).

These approaches are not identical to the ChartDQA benchmark itself, but they illustrate the principal inductive biases that ChartDQA is designed to probe: explicit decomposition of reasoning steps, stronger chart-component grounding, and tighter alignment between visual marks and underlying numeric structure (Zhang et al., 2024, Zheng et al., 2024, Chen et al., 21 Jul 2025).

7. Difficulty profile and future directions

Evidence from neighboring benchmarks indicates that low-level and data-grounded chart understanding remains difficult for current multimodal models. On the notebook-derived analytical-narrative benchmark, GPT-4.1 achieved 69.27%, Qwen2.5-VL-32B achieved 56.59%, and Qwen2.5-VL-7B achieved 31.71%; literal tasks such as Retrieve Value and Determine Range were handled best, while higher-order tasks such as Make Comparisons and Find Anomalies remained challenging (Hutchinson et al., 2 Jul 2025). The same work notes that humans on similar tasks typically exceed 90–95% accuracy, and interprets uneven model behavior across reasoning types as evidence of superficial pattern matching rather than deep chart understanding (Hutchinson et al., 2 Jul 2025).

ChartInsights provides a second difficulty signal. Across 19 advanced MLLMs evaluated on low-level chart tasks, the average accuracy was 39.8%, with GPT-4o at 69.17%. Structured prompting materially changed results: Chain-of-Charts raised GPT-4V from 59.0% to 80.49%, and combining it with visual prompting reached 83.8% (Wu et al., 2024). This suggests that chart reasoning performance is sensitive not only to model capacity but also to decomposition strategy and attention guidance.

Within the ChartDQA benchmark specifically, the stated research implications are data-grounded multimodal chain-of-thought, robust chart-data alignment, and open-domain chart analysis. More broadly, the surrounding survey literature identifies realism in datasets, robust chart data extraction, complex and compositional reasoning, open-vocabulary and explanatory answers, multimodal and conversational QA, modeling chart semantics with large pretrained transformers, and accessibility as open challenges for chart question answering (Fan et al., 18 Jul 2025, Hoque et al., 2022).

A common misconception is that broader chart-type coverage alone is sufficient for chart understanding. The current literature does not support that view. Synthetic coverage, gold raw data, and explicit alignment tasks address one set of limitations; ecologically valid narratives, multi-view interaction, and document-level layout analysis address others. ChartDQA is best understood as a benchmark that sharpens one particularly important frontier: whether a model can align chart imagery with the underlying data representation closely enough to support literal extraction, inferential computation, and higher-order reasoning within a single evaluation framework (Fan et al., 18 Jul 2025, Hutchinson et al., 2 Jul 2025).