PolyChartQA: Multilingual & Multi-chart Benchmark

• PolyChartQA is a benchmark suite that extends chart question answering by incorporating multilingual evaluation and composite multi-chart figures.
• The multilingual benchmark features 26,151 QA pairs from 22,606 charts across 10 languages and 16 chart types, utilizing a structured JSON-plus-code pipeline for quality control.
• The multi-chart benchmark evaluates sub-chart localization and cross-panel reasoning using composite figures from recent computer science papers, highlighting challenges in realistic multi-view analysis.

Searching arXiv for the cited PolyChartQA and CQA survey papers to ground the article in current literature. arXiv search query: (Xu et al., 16 Jul 2025) PolyChartQA Benchmarking Large Vision-LLMs with Multilingual Chart Question Answering PolyChartQA denotes two distinct chart question answering benchmarks that extend the CQA problem beyond the single-chart, English-centric setting emphasized in much earlier work. One benchmark targets multilingual chart understanding across 10 languages and 16 chart types, with 22,606 charts and 26,151 question-answering pairs; the other targets reasoning over real composite multi-chart figures, with 534 multi-chart images, 2,297 sub-charts, and 2,694 QA pairs. Within the broader CQA landscape, both benchmarks instantiate directions identified as underexplored in the survey literature: realistic diversity, multilingual labels, and multi-view composition (Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026, Hoque et al., 2022).

1. Terminological scope and benchmark identity

The name PolyChartQA is currently used for two separate resources rather than a single unified dataset. The 2022 CQA survey does not explicitly mention PolyChartQA, nor variants such as “Poly-ChartQA,” but its taxonomy is useful for situating both later benchmarks: one expands CQA along the multilingual axis, and the other expands it along the multi-chart axis (Hoque et al., 2022).

Benchmark	Primary focus	Scale
PolyChartQA (Xu et al., 16 Jul 2025)	Multilingual chart QA	22,606 charts; 26,151 QA pairs; 10 languages; 16 chart types
PolyChartQA (Efat et al., 23 Apr 2026)	QA over multi-chart images	534 images; 2,297 sub-charts; 2,694 QA pairs

This suggests that “PolyChartQA” functions less as a single canonical benchmark name than as a label for two complementary attempts to move chart QA beyond narrow single-view settings. In the multilingual benchmark, “poly” is operationalized through linguistic breadth and chart-type diversity. In the multi-chart benchmark, it is operationalized through composite figures, cross-panel reasoning, and sub-chart localization (Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026).

2. Position within the chart question answering landscape

Chart question answering systems, in the survey’s formulation, typically receive an image $I$ of a chart, a natural-language question $q$, and optional metadata $M$ such as OCR tokens, chart structure, data tables, Vega-Lite specifications, or element bounding boxes. The survey formalizes three common views of the task: end-to-end prediction $a = f(I, q, M)$; parse-then-reason pipelines in which $D = parse(I)$ and then $a = g(D, q)$; and program-execution settings in which $a = p(D)$, where $p$ is a logical or semantic program derived from the question (Hoque et al., 2022).

The same survey organizes CQA by question intent and reasoning demand: lookup or retrieval, comparison, aggregation, trend and slope, extremum, arithmetic, and textual reasoning; it also distinguishes visual from non-visual questions and simple from compositional questions. Against that taxonomy, the multilingual PolyChartQA emphasizes multilingual robustness under diverse chart types, while the multi-chart PolyChartQA emphasizes localization of the relevant sub-chart, legend and label transfer across panels, and cross-chart comparison or aggregation without explicit “first chart/second chart” cues (Hoque et al., 2022, Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026).

Earlier datasets summarized in the survey were often synthetic, template-driven, or restricted to single chart types. The survey explicitly identifies realism, multilingual charts, and multi-view questions as open challenges. The two PolyChartQA benchmarks can therefore be read as concrete responses to those gaps: one addresses multilingual chart rendering and evaluation, and the other addresses question answering over real composite figures rather than isolated plots (Hoque et al., 2022).

3. Multilingual PolyChartQA

The multilingual PolyChartQA benchmark is presented as the first large-scale multilingual chart question answering benchmark. It covers 22,606 chart images and 26,151 QA pairs across 10 languages—English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese—and 16 chart types: 3d-bar, area, bar, box, bubble, candlestick, funnel, heatmap, histogram, line, multi-axes, pie, radar, rings, rose, and treemap (Xu et al., 16 Jul 2025).

Its construction begins from English seed corpora selected from ChartQA and ChartX. Cleaning and validation include an automatic QA consistency check using Gemini 2.5 Pro-exp-0325, which removed approximately 17–20% of noisy or unanswerable triplets, followed by answer normalization to concise forms and human review on 10% of the cleaned set, which achieved a pass rate above 98%. Each cleaned chart is then decoupled into a structured JSON specification encoding data table, chart type, layout, and text, together with an executable Plotly Python script that reads the JSON and renders the chart. This separation of chart data from rendering code is the benchmark’s central engineering idea: multilingual charts can be regenerated by translating only user-facing strings while reusing the rendering program (Xu et al., 16 Jul 2025).

The multilingual expansion uses joint LLM translation so that chart JSON and associated QA pairs are translated together in a single structured prompt. Keys, numerical values, color codes, and structural fields remain unchanged. Quality control combines back-translation with METEOR and an LLM judge, retaining only samples with METEOR greater than 0.6 and a Gemini semantic-alignment score of at least 4/5. On an English–Chinese subset of 250 samples, human averages were 2.904 for image quality, 2.948 for QA correctness, and 2.908 for translation accuracy on a 3-point scale; the Gemini proxy achieved 2.956, 2.984, and 2.892 respectively, with exact-match agreement of 91.2% for image quality, 94.4% for QA, and 90.8% for translation (Xu et al., 16 Jul 2025).

Evaluation uses relaxed accuracy following PlotQA and ChartQA. Numeric answers are treated as correct within 5% tolerance, while non-numeric answers require exact string match:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\Big[ \big(\text{isnum}(y_i) \wedge |\hat{y}_i - y_i| \le 0.05\cdot |y_i|\big) \;\vee\; \big(\neg\text{isnum}(y_i) \wedge \hat{y}_i = y_i\big) \Big].$$

Macro-averages are reported both with English and without English. No external OCR tools are used; models read text directly from chart images. Inference is zero-shot unless otherwise stated, with temperature set to 0.01 and a uniform prompt instructing the model to answer using a word or phrase in the target language or a number in digits (Xu et al., 16 Jul 2025).

The reported results show a substantial multilingual gap. Gemini-2.5-Pro is best overall with average relaxed accuracy of 0.685 with English and 0.682 without English. GPT-4o reaches 0.509 and 0.502. The best open-source result is Qwen2.5-VL-7B at 0.538 and 0.530. Performance declines are especially marked for low-resource languages and non-Latin scripts. Few-shot prompting does not reliably close the gap: for Qwen2.5-VL-7B, 0-shot yields 0.538 average with English, 4-shot yields 0.559, and 8-shot yields 0.560, with gains described as modest and inconsistent. Cross-lingual transfer experiments further show that language consistency across modalities matters: localized chart plus translated question (“Native”) performs better than English chart plus translated question (“Img”) or localized chart plus English question (“QA”) (Xu et al., 16 Jul 2025).

4. Multi-chart PolyChartQA

The multi-chart PolyChartQA benchmark is designed for question answering over real multi-chart figures rather than individual plots. It contains 534 multi-chart images with a total of 2,297 sub-charts, all sourced from 168 open-access computer science papers published in 2024 across EMNLP, SIGCOMM, ICSE, and ICML. The dataset includes 11 chart types—Bar, Line, Scatter, Area, Pie, Histogram, Box, Violin, Point, Spider, and Surface—with a median of 3 sub-charts per image and a maximum of 72. Of the images, 85.58% are homogeneous in chart type and 14.42% are mixed-type. The QA set contains 2,694 open-ended pairs: 519 human-authored and 2,175 MLM-generated and manually verified (Efat et al., 23 Apr 2026).

The benchmark is explicitly motivated by the observation that real analytical workflows often require interpreting multiple related charts together. The task therefore requires localizing the relevant sub-chart or sub-charts before reasoning, aligning legends and labels across panels, transferring symbol or category semantics from one panel to another, and conducting cross-chart comparison or aggregation under potentially differing scales, axes, or visual encodings. Questions are written without explicit chart references such as “left” or “right,” and the dataset annotates sub-chart count, sub-chart chart types, image-level homogeneity, question difficulty, and question type (Efat et al., 23 Apr 2026).

Data collection begins with figure extraction from PDFs using PDFFigures 2.0, followed by manual filtering to retain only multi-chart images. Structural annotations were produced by the first author and independently checked by GPT-4o across all 534 images, yielding 91.95% agreement; disagreements were 4.49% for sub-chart count, 3.93% for sub-chart type, and 0.75% for homogeneity. Human-authored questions were written over 238 images, then validated by the second and third authors; one image and its QA were removed and 98 QAs revised, producing 519 final human-authored QAs over 237 images. MLM-generated questions were produced by GPT-4.1, which generated 4,308 candidates; manual verification removed 2,133 QAs, or 49.5%, and 3 images. Removal criteria included explicit chart position cues, partial or incorrect QAs, caption-derived questions, multiple-valid-answer ambiguity, symbol or format errors, hallucinations, instruction failures, and reliance on external knowledge (Efat et al., 23 Apr 2026).

Evaluation uses H-Accuracy, L-Accuracy, and BERTScore. H-Accuracy is human-evaluated exact or semantic correctness on human-authored questions. L-Accuracy is determined by an LLM judge that compares a model answer to the ground truth and outputs correct or incorrect. Among Claude-3.7-Sonnet, GPT-4.1, and Gemini-2.0-flash, Claude-3.7-Sonnet was selected as the judge because it had the highest agreement with human annotation, with Cohen’s $\kappa$ ranging from 0.73 to 0.92. Unlike the multilingual benchmark, the paper does not formalize case normalization, unit normalization, or explicit numeric tolerance thresholds; semantic equivalence is delegated to human or LLM judgment (Efat et al., 23 Apr 2026).

The central empirical result is that moving from single charts to real multi-chart compositions substantially degrades model performance even when the question and answer remain identical. In the MultiChartQA-RQ1 experiment, L-Accuracy drops by 26.85% to 36.98% for closed-source models, 3.56% to 36.44% for open-source models, and 4.66% to 6.30% for chart-specific models; BERTScore drops by up to 12.38%. Human-authored questions are also markedly harder than MLM-generated questions: the reported gap reaches 27.4% L-Accuracy, exemplified by GPT-4.1 under Chain-of-Thought, where performance is 0.5954 on human-authored questions versus 0.8694 on MLM-generated questions (Efat et al., 23 Apr 2026).

Question difficulty and type materially affect results. Performance declines monotonically with difficulty level, and data retrieval is the hardest question type, with Structural $q$0 Reasoning $q$1 Data Retrieval. Accuracy also tends to decrease as the number of sub-charts increases, and many models perform better on non-homogeneous than homogeneous figures, although the paper notes exceptions including GPT-4.1, Gemma, and ChartGemma. To improve performance, the paper proposes VDSP, a three-stage prompting strategy—Chart Structuring and Question Decomposition, Structured Reasoning Execution, and Self-Verification—which improves L-Accuracy by up to 5.39% over zero-shot and up to 4.62% over Chain-of-Thought. For Claude-3.7-Sonnet on human-authored questions, L-Accuracy rises from 0.6570 in zero-shot to 0.6647 with Chain-of-Thought and 0.7110 with VDSP (Efat et al., 23 Apr 2026).

5. Common technical challenges and failure modes

The multilingual and multi-chart PolyChartQA benchmarks stress different capabilities, but their error profiles intersect around text reading, structural grounding, and multi-step reasoning. The CQA survey identifies OCR inaccuracies, misreading axes or ticks, scale conversions, visual encoding ambiguities, aggregation and composition errors, multi-chart alignment failures, domain and style shifts, and annotation ambiguity as recurring failure modes across chart QA systems (Hoque et al., 2022).

In the multilingual benchmark, the most salient issues are script handling and typography, especially for Arabic, Bengali, Urdu, and Japanese; right-to-left text; locale-specific numerals and units; and cross-lingual mismatch between the language of the chart and the language of the question. Because no external OCR is used, the model must read fine-grained text directly from the chart image and then reason over the visual structure. The paper reports that numeric or aggregation operations and extremum questions are particularly sensitive to small text-reading errors in non-Latin scripts, and that the high translation-quality scores imply the main bottleneck is model limitation rather than dataset noise (Xu et al., 16 Jul 2025).

In the multi-chart benchmark, manual analysis of one-third of the human-authored questions that all models answered incorrectly identified several recurring failure modes: line selection errors in line plots at 21–26%, visual value misreading at 15–26%, and sub-chart misidentification at 15–21%. Other errors include axis misalignment, arithmetic mistakes, legend association errors, multi-step reasoning failures, incorrect comparisons, and incomplete answers from open-source models. These findings align closely with the survey’s earlier warning that multi-chart alignment, legend-to-mark association, and compositional numerical reasoning remain weak points in CQA (Efat et al., 23 Apr 2026, Hoque et al., 2022).

6. Significance, limitations, and future directions

Taken together, the two PolyChartQA benchmarks extend chart QA in directions that the survey explicitly describes as important for the field’s next stage: broader chart diversity, multilingual support, realistic real-world charts, multi-view reasoning, and evaluation protocols that go beyond simple exact match. The multilingual benchmark supplies a decoupled JSON-plus-code pipeline for reproducible multilingual rendering with rigorous quality control, while the multi-chart benchmark supplies real composite figures, question difficulty labels, question type labels, and structural annotations for sub-chart count and homogeneity. This suggests that later CQA work will need to combine linguistic robustness, structural grounding, and cross-panel reasoning rather than treating them as separate problems (Hoque et al., 2022, Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026).

Both benchmarks are intentionally limited in ways that delimit their immediate scope. The multilingual benchmark is test-only, covers 10 languages rather than the full set of low-resource languages, and notes that rare translation or font-rendering errors may persist despite strong quality control. The multi-chart benchmark is also presented as an evaluation benchmark without train/validation/test splits in the paper text; its human-authored QA component is relatively small at 519 questions, and all figures come from 2024 computer science papers, with 85.58% homogeneous chart-type compositions (Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026).

The future directions reported in the papers and survey are correspondingly concrete. For multilingual PolyChartQA, future work includes multilingual chart summarization, fact-checking, and trend analysis, as well as targeted multilingual chart training and more robust text reading across scripts. For multi-chart PolyChartQA, future work includes broader domain coverage beyond computer science, more heterogeneous compositions, and training strategies or model architectures that better localize and reason across sub-charts. The survey further recommends structured annotations, program or rationale supervision, explainability through evidence and executable programs, robust normalization and evaluation under OCR noise, and support for multi-chart narratives and conversational context. In that sense, PolyChartQA is best understood not as a closed benchmark family but as an indicator of where chart question answering research is moving: from synthetic, single-view, predominantly English settings toward realistic, diverse, and structurally grounded chart reasoning (Xu et al., 16 Jul 2025, Efat et al., 23 Apr 2026, Hoque et al., 2022).