BigCharts: Chart Understanding Pipeline

• BigCharts is a chart-understanding framework that re-plots real-world charts to preserve visual authenticity and ensure accurate underlying data.
• It overcomes limitations in synthetic datasets by addressing issues such as homogeneous styles and noise from automated data recovery.
• BigCharts-R1 enhances performance by combining chain-of-thought supervision with GRPO-based reinforcement learning for precise numeric reasoning.

BigCharts is a chart-understanding dataset creation pipeline, and BigCharts-R1 is the chart reasoning model produced from it through a two-stage post-training strategy that combines supervised finetuning with Group Relative Policy Optimization (GRPO)-based reinforcement learning. The central objective is to close three specific deficiencies in chart question answering: visually homogeneous synthetic training corpora, inaccurate underlying data recovered from real charts, and an exclusive dependence on supervised finetuning that tends to overfit and generalize poorly to out-of-distribution chart styles. BigCharts addresses these issues by replotting real-world-inspired charts so that stylistic authenticity is retained while accurate underlying code and structured data are available for annotation; BigCharts-R1 adds chain-of-thought supervision and chart-specific verifiable rewards to improve robustness, numeric precision, and reasoning generalization across diverse chart styles and domains (Masry et al., 13 Aug 2025).

1. Problem setting and motivation

The work is positioned against two dominant paradigms in chart QA. First, purely synthetic corpora such as FigureQA, DVQA, and PlotQA are generated with restricted plotting parameters, lack real-world design variety, and often use templated question styles. The reported consequence is that models trained on these datasets overfit to narrow layout distributions and fail on authentic charts with heterogeneous fonts, palettes, gridlines, legends, and axis formats (Masry et al., 13 Aug 2025).

Second, pipelines that crawl real chart images and then recover their underlying tables or values automatically introduce significant estimation errors. In the reported framing, those errors propagate into training targets, leading to inaccurate QA labels and degraded downstream performance. The paper also argues that most chart-specific vision-LLMs rely solely on supervised finetuning over such low-quality corpora, limiting robustness and generalization.

Within this formulation, BigCharts is intended to preserve real-world visual authenticity while restoring data fidelity, and BigCharts-R1 is intended to exploit that improved supervision by combining chain-of-thought chart reasoning with GRPO-based reinforcement learning. A plausible implication is that the contribution is not merely larger-scale data collection, but a redefinition of the supervision source: from either synthetic charts or noisy reverse-engineered real charts to replotted, code-backed charts that align image, code, and question-answer annotations.

2. Dataset construction and replotting pipeline

BigCharts begins from a heterogeneous image pool assembled from aggregated datasets, Common Crawl via Mint-1T PDFs, and targeted Google Search crawling. The aggregated datasets are ChartGemma, FigureQA, DVQA, PlotQA, and ArXivQA, totaling 174,277 images. The Common Crawl stage uses a two-stage filtering process with a ResNet-50 chart classifier trained on labeled charts and negatives from CC-12M and ImageNet; 5,000 samples are manually re-labeled to improve precision, yielding 57,196 validated charts. A targeted Google Search crawl based on 200 curated keywords contributes 13,941 charts spanning heatmaps, scatter plots, dashboards, infographics, and related forms. The combined initial pool contains 245,414 chart images before replotting, and licensing details and usage terms are provided on the project website, bigcharts.github.io (Masry et al., 13 Aug 2025).

The defining operation is the replotting process for accurate underlying data. Given a real chart image, Gemini Flash 2.0 is used to generate plotting code that replicates the chart’s design and content. New chart images are then rendered from the generated code, and the original images are discarded in order to avoid data inaccuracies and ensure metadata consistency. The implementation primarily uses matplotlib and plotly for most charts, with Chart.js reserved for stylistically complex charts, especially some from ChartGemma and Google. Any generated code that fails to render is excluded. The result is 134,950 replotted charts with accurate underlying code and structured data.

Fidelity constraints explicitly require matching color schemes, layout, axis labels and formats, legends, titles, and fonts. The supported space includes multi-series charts such as multi-line and grouped or stacked bar charts, as well as scatter, area, pie and donut, and other specialized chart types. The methodological claim is that BigCharts bridges realism and accuracy by replotting real-world-inspired styles with exact underlying data, then generating QA from both image and code. This suggests that the pipeline treats chart understanding as a joint visual and programmatic reconstruction problem rather than a purely OCR- or detection-driven problem.

3. Diversity, authenticity, and annotation schema

The dataset’s authenticity is attributed to real-world inspiration drawn from Mint-1T PDFs and targeted Google queries, which yield styles beyond standard plotting defaults. Diversity is quantified through automatic chart-type classification and topic clustering. The topic distribution is reported as Finance/economy 39%, STEM 29.7%, and Data Science 27.5%. The chart-type distribution spans bar, line, multi-line, scatter, stacked bars, and area, while the “Other” category includes heatmaps, radar, waterfall, violin, and related forms (Masry et al., 13 Aug 2025).

For each chart, BigCharts generates 16 questions with chain-of-thought reasoning. The question families cover direct data retrieval, visual analysis, numerical computations, yes/no numerical questions, counting tasks, multiple-choice questions with variable option formats, unanswerable questions with “Unanswerable” as ground truth, and conversation-style prompts in which multi-turn context precedes a final question. Each sample contains a question, a final answer, and chain-of-thought reasoning.

The answer distribution is skewed toward numeric responses. Textual answers account for approximately 342k instances, Yes/No answers for approximately 141k, and multiple-choice answers for approximately 118k; unanswerable cases are explicitly included. Chain-of-thought sequences have median length 39 tokens, average length 49.65 tokens, and maximum length 1,933 tokens. In aggregate, the dataset contains 134,950 re-rendered, code-backed charts and approximately 1.8 million QA pairs. The paper does not specify resolution or multilingual composition, and supervised finetuning uses the full corpus.

A recurrent comparison is made to prior chart datasets. DVQA, PlotQA, and FigureQA are described as large-scale but synthetic, with limited visual diversity, templated questions, and often missing explicit numeric labels, thereby requiring interpolation and exhibiting low realism. ChartQA and CharXiv are characterized as visually more realistic but potentially lacking high-fidelity underlying data and comprehensive reasoning supervision. ChartGemma and TinyChart are described as instruction-tuning pipelines that either rely on datatable generation from images via closed models, thereby introducing estimation errors, or employ program-of-thought formats without guaranteed visual fidelity. BigCharts is therefore positioned as an overview of realistic style and exact underlying data.

4. Training framework: supervised finetuning and GRPO-based RL

BigCharts-R1 uses Qwen2.5-VL-Instruct backbones in 3B and 7B scales without architectural modifications. Multimodal fusion and tokenization follow the default Qwen2.5-VL design. The first stage is supervised finetuning on chain-of-thought plus final answer sequences conditioned on chart image and prompt. The objective is sequence-level maximum likelihood:

$$L_{\mathrm{SFT}}(\theta) = - \mathbb{E}_{(x, I, y)} \left[\sum_{t=1}^{T} \log \pi_\theta(y_t \mid y_{<t}, x, I)\right],$$

where $x$ is the text prompt, $I$ the chart image, and $y$ the target chain-of-thought-plus-answer sequence (Masry et al., 13 Aug 2025).

The reported supervised schedule is one epoch over BigCharts with learning rate $2 \times 10^{-5}$, batch size 32, warmup ratio 0.1, and a cosine scheduler, implemented with LlamaFactory. The use of the full corpus for SFT reflects the paper’s emphasis on broad stylistic and topical coverage rather than narrow benchmark alignment.

The second stage is reinforcement learning with verifiable rewards (RLVR), using the SFT model as the reference policy $\pi_{\mathrm{ref}}$. The RLVR objective is

$$J(\theta) = \mathbb{E}_{x}\,\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)} [R(x,y)] - \beta \,\mathbb{E}_{x}\, \mathrm{KL}(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid x)).$$

Optimization is performed through Group Relative Policy Optimization. For a prompt $x$, a group of $G$ responses $\{o_i\}_{i=1}^{G}$ is sampled from the current policy. Per-response rewards are $x$0. The group-relative advantage is

$x$1

where $x$2 is the mean and $x$3 the standard deviation over the group rewards. The policy update maximizes

$x$4

The paper states that GRPO avoids a learned critic, thereby reducing memory and improving efficiency.

The chart-specific reward design contains two components. The first is the numeric error and smooth reward, denoted CERM. The relative numeric error is

$x$5

The reward is then defined as

$x$6

The second component is a response format reward:

$x$7

The total reward is

$x$8

The RL stage uses 22,297 chart images and 32,297 questions from human-labeled or verified sources: the ChartQA training split plus 1,000 samples from PlotQA (v1/v2), DVQA, and FigureQA training subsets with verifiable answers. Training runs for one epoch with batch size 8, learning rate $x$9, and $I$0 candidates per prompt on a single node with $I$1 H100-80GB GPUs.

5. Evaluation results and empirical profile

Evaluation spans both real-world and synthetic benchmarks. Real-world evaluation uses ChartQA with relaxed accuracy and CharXiv with reasoning and descriptive scores obtained via GPT-4o using the official prompt. Synthetic evaluation uses FigureQA-Sub and DVQA-Sub with exact accuracy, and PlotQA-Sub with relaxed accuracy. Each synthetic benchmark is reduced to a 1,000-QA-pair subset for tractable evaluation while maintaining diversity (Masry et al., 13 Aug 2025).

The principal reported results are as follows.

Model	Average across tasks	Selected notes
BigCharts-R1-3B	72.14	ChartQA avg 88.16; CharXiv 37.40 reasoning, 62.38 descriptive
BigCharts-R1-7B	74.48	ChartQA avg 89.84; CharXiv 41.30 reasoning, 66.58 descriptive
Qwen2.5-VL-3B + SFT	68.71	SFT-only baseline
Qwen2.5-VL-7B + SFT	71.66	SFT-only baseline
Intern-VL2.5-8B	70.41	Open-source baseline
GPT-4o	61.22	Closed-source baseline

At the benchmark level, BigCharts-R1-3B attains 80.10 and 81.00 on FigureQA-Sub Val1 and Val2, 81.20 and 80.60 on DVQA-Sub ValE and ValH, 78.50 and 59.90 on PlotQA-Sub T1 and T2, and 94.32, 82.00, and 88.16 on ChartQA aug, hum, and avg, respectively. On CharXiv it reaches 37.40 for Reasoning and 62.38 for Descriptive. BigCharts-R1-7B improves these figures to 81.20 and 81.20 on FigureQA-Sub, 83.80 and 83.60 on DVQA-Sub, 80.90 and 61.90 on PlotQA-Sub, 94.88, 84.80, and 89.84 on ChartQA, and 41.30 and 66.58 on CharXiv Reasoning and Descriptive.

The paper reports several relative gains. BigCharts-R1-3B exceeds the SFT-only 3B model by 3.43 average points, surpasses Intern-VL2.5-2B by 7.55 points, and outperforms Intern-VL2.5-8B by 1.73 points despite its smaller size. BigCharts-R1-7B exceeds Intern-VL2.5-8B by 4.07 average points and is described as competitive with closed-source models on ChartQA and the synthetic subsets.

Ablation studies isolate the role of the dataset and the replotting step. Under the same Qwen2.5-VL-3B backbone and hyperparameters, finetuning on TinyChart data yields an average of 48.80, finetuning on ChartGemma data yields 58.05, and finetuning on BigCharts yields 68.70. Training on QA generated from original crawled charts without code yields 66.22 average, whereas training on QA from BigCharts replotted charts with code yields 68.70. The stated conclusion is that replotting with code improves QA accuracy and downstream performance by enabling image-and-code-consistent QA generation.

The RL versus SFT comparison specifically targets out-of-distribution generalization. When trained only on ChartQA, with chain-of-thought generated via Gemini Flash because ChartQA does not originally contain CoT, GRPO consistently outperforms SFT on both in-distribution ChartQA and out-of-distribution DVQA-Sub, PlotQA-Sub, and FigureQA-Sub. The paper emphasizes especially notable gains on OOD synthetic sets where numeric interpolation and visual reasoning are required.

6. Reasoning behavior, limitations, and future directions

The qualitative analysis presents a concrete account of the model’s chart reasoning behavior. Reported chain-of-thought traces show stepwise identification of visual primitives and metadata, such as reading y-axis units and tick scales, retrieving values for specific series and positions, and then computing differences, means, sums, or ratios. The model is described as grounding series and category names in axes and legends, selecting relevant bars or points from positional descriptors such as leftmost or highest peak, and producing concise answers after arithmetic. In this interpretation, BigCharts-R1 is not limited to extraction; it is designed for visually grounded symbolic reasoning over chart structure (Masry et al., 13 Aug 2025).

The reward design is directly tied to those behaviors. CERM gives partial credit for numerically close answers, which is reported to guide the model in refining interpolations when exact explicit labels are absent. The response format reward enforces strict answer formatting, including bare numerals, list formatting for multiple items, and no units, thereby reducing post-processing errors. The paper further argues that BigCharts-R1 handles charts lacking explicit numeric labels better than prior systems because CERM rewards numeric proximity and the SFT stage is trained on image-and-data-consistent supervision.

The error analysis identifies several failure modes: misreading crowded legends or occluded axis labels, over-smoothing at multi-line intersections, and confusion under unusual color palettes. Charts without explicit labels still require visual estimation; reinforcement learning helps, but extreme occlusions remain difficult. An important qualification is that on CharXiv’s Descriptive subset, RL slightly underperforms SFT. The reported explanation is that GRPO rewards emphasize reasoning and numeric precision rather than verbatim retrieval. This addresses a common misconception that reinforcement learning uniformly improves every dimension of chart understanding; in the paper’s empirical profile, gains are concentrated in reasoning and numeric tasks.

The limitations section also notes that, although visually diverse, the dataset remains dominated by common chart types such as bar, line, and pie, and by topical clusters such as finance and economy. Rare genres, including intricate dashboards and unconventional infographics, may therefore be underrepresented. Replotting quality depends on code generation quality, failed renders are discarded, and subtle style nuances may be imperfectly captured. The numeric relative-error reward assumes a well-defined ground-truth numeric target, making free-form textual explanations or multi-answer rationales less directly optimized. Web-sourced charts may also reflect biases in source domains and topics; usage guidance and licensing terms are provided on the project website.

The stated future directions are to extend the methodology to table reasoning and geometric figures, to design richer task-aligned reward functions for chart summarization, fact-checking, and broader visual inference, and to strengthen backbone vision pretraining for improved chart parsing and OCR robustness. This suggests that the BigCharts framework is intended as a general recipe for visually grounded reasoning tasks in which a high-fidelity programmatic reconstruction can serve as a scaffold for both supervision and verifiable reinforcement signals.