ChartScope: Chart-Specialized LVLM Framework

• ChartScope is a chart-specialized large vision-language model framework designed for comprehensive chart understanding by aligning visual cues with structured JSON data.
• It uses a scalable synthetic data pipeline generating millions of chart images with corresponding JSON pairs to overcome limitations of generic image-language pretraining.
• Its Dual-Path training strategy, combining data-driven and JSON-only QAs, enhances chart-data extraction and reasoning, leading to improved performance on benchmarks like ChartDQA.

ChartScope is a chart-specialized large vision-LLM customization framework for comprehensive chart understanding. It is designed around two linked objectives: breadth, meaning coverage of many chart types beyond standard bar, line, and pie charts, and depth, meaning recovery of and reasoning over the chart’s underlying data rather than reliance on visible text alone. The framework addresses what it describes as a vision-language alignment gap between generic natural-image pretraining and the structured semantics of digital charts, and it combines a large synthetic data pipeline, chart-specific pretraining on chart-description and chart-JSON pairs, a Dual-Path training strategy, and a new benchmark, ChartDQA (Fan et al., 18 Jul 2025).

1. Concept and scope

ChartScope is not presented as a wholly new multimodal architecture from scratch. Rather, it is an LVLM customized and pre-trained for chart understanding, instantiated on top of LLaVA-family models and TinyLLaVA (Fan et al., 18 Jul 2025). Its central premise is that chart understanding cannot be reduced to ordinary image captioning or OCR-heavy visual question answering, because charts are renderings of structured data together with titles, axes, scales, legends, and visual encodings.

The framework is motivated by two limitations that the authors identify in prior chart-focused LVLM customization. First, existing methods often rely on paired data from only a small number of chart types, which limits generalization. Second, they typically lack explicit pre-training for chart-data alignment, so models may answer by exploiting visible labels or annotations rather than learning the mapping from graphical marks to values (Fan et al., 18 Jul 2025).

This framing places ChartScope within a broader effort to treat chart understanding as a problem of structured multimodal reasoning rather than generic image-language matching. A plausible implication is that its intended contribution is less a single-task performance gain than a redefinition of what counts as chart comprehension: not only recognizing a plotted pattern, but also aligning the image with the structured data it encodes.

2. Training architecture and workflow

ChartScope uses a three-stage training procedure. The stages and their roles are explicit.

Stage	Data	Role
1. Projector pre-training	Original LLaVA pretraining data, synthetic chart-description pairs, synthetic chart-JSON pairs	Visual-language alignment, especially chart-data alignment
2. End-to-end fine-tuning	Original LLaVA Instruct-150K, synthetic chart QA, Dual-Path augmented QA forms	Preserve generic reasoning while specializing for charts
3. Downstream adaptation	LoRA for LLaVA variants; standard fine-tuning with TinyChart for TinyLLaVA-based variant	Benchmark-specific adaptation

The reported model variants are LLaVA-7B, LLaVA-13B, and TinyLLaVA-3.1B@768 (Fan et al., 18 Jul 2025). The strongest reported variant on several chart benchmarks is the TinyLLaVA-based version, which the paper interprets as evidence that training strategy and data quality matter at least as much as raw parameter count.

A recurrent misconception would be to treat ChartScope as merely instruction-tuned chart QA. The paper explicitly describes a more specific workflow. Stage 1 includes synthetic chart-JSON pairs, not only chart descriptions, so the pretraining target already emphasizes structured chart semantics. Stage 2 mixes standard chart QAs with two augmented forms introduced by the framework’s Dual-Path strategy. At inference time, the model can also use data prompting, in which it is asked first to extract raw data and then answer the question; the paper reports that this improves downstream performance (Fan et al., 18 Jul 2025).

3. Synthetic data generation and supervision

A major part of ChartScope is its scalable synthetic data pipeline. The authors describe it as quadratic-scale data generation, because it separates data generation from code generation. If there are $M$ data files and $N$ code files for a chart type, any compatible data file can be paired with any compatible plotting script, yielding approximately $N \times M$ image realizations (Fan et al., 18 Jul 2025).

For each chart type, a “JSON expert” LLM first creates a JSON template and a README. The template specifies chart structure, including items such as title, x-axis information, y-axis information, and raw data. The README explains chart type definition, key semantics, and expected structure. A “data expert” LLM then generates JSON data files from the chart type, template, README, and a sampled topic. Separately, a “code expert” LLM generates Python plotting scripts with variation in color, legend, grid, font, mark texture, and related style factors (Fan et al., 18 Jul 2025).

The paper states that the pipeline considers 20 different chart types. For each type it generates $N = 400$ Python codes and $M = 1000$ JSON data files. After filtering by file-structure correctness, Python execution success, and OCR-based checks, the pipeline yields approximately 5 million images. Around 2 million synthetic pairs are used for the 13B model and around 500k pairs for the 3B model (Fan et al., 18 Jul 2025).

The supervision is correspondingly dense. For each chart image, the stored artifacts include the rendered chart image, raw JSON data, the README, the Python script, and 17 general QA pairs: 1 description QA, 1 summary QA, 5 literal QAs, 5 inferential QAs, and 5 reasoning QAs (Fan et al., 18 Jul 2025). JSON is preferred over CSV because it can encode titles, axis descriptions, scales, and more complex chart structures.

This design differs in emphasis from prior synthetic chart corpora that focus primarily on image-text or image-table pairs. Related work such as SynChart likewise stresses large-scale synthetic chart generation, but SynChart is centered on approximately 3.93M chart images and over 75M annotations, whereas ChartScope’s pipeline is organized specifically around chart-JSON alignment and Dual-Path QA training (Liu et al., 2024).

4. Dual-Path training and chart-data alignment

The framework’s second major idea is Dual-Path training. The paper presents chart QA as a task that depends on two distinct abilities: recovering the chart’s underlying structured data and reasoning over that data. Dual-Path training therefore introduces two complementary supervision routes (Fan et al., 18 Jul 2025).

The first route is Data-driven QAs. These are multi-turn QAs in which the model is prompted to extract the chart’s raw JSON data from the image and then answer the question using the extracted JSON and the chart. This path is intended to strengthen chart-to-data grounding.

The second route is JSON-only QAs. These are text-only QAs in which the image is replaced by the raw JSON data and the README. This path trains reasoning directly over structured chart data and is intended to preserve the base LLM’s reasoning capacity while specializing it for chart semantics (Fan et al., 18 Jul 2025).

The implied factorization is:

$$\text{chart image} \rightarrow \text{structured data} \rightarrow \text{reasoned answer}$$

The paper does not provide an explicit new loss formula for this decomposition; instead, it describes the objective conceptually as standard autoregressive pretraining and instruction-tuning losses over chart-description pairs, chart-JSON pairs, general chart QAs, JSON-only QAs, and Data-driven QAs (Fan et al., 18 Jul 2025).

The ablations are consistent with the alignment claim. On ChartQA, pretraining with original LLaVA-CC3M only yields 44.80 on the human split; adding chart-description pairs raises this to 48.56; adding chart-JSON pairs raises it to 52.28. In fine-tuning ablations, adding JSON-only QAs and Data-driven QAs improves the human split further, and Data Prompting at inference raises it to 56.96 (Fan et al., 18 Jul 2025). This suggests that the chart-data alignment objective is not incidental but central to the framework’s gains.

5. ChartDQA and empirical performance

ChartScope is evaluated together with a new benchmark, ChartDQA, which is intended to test comprehensive chart understanding rather than only short-form QA on heavily annotated charts. ChartDQA contains 5.48k images, covers 20 chart types, provides 13.5 average QAs per image, supports multi-level QA, includes raw data per image, and includes chart style variation. The benchmark is human filtered for Answerability and Correctness (Fan et al., 18 Jul 2025).

ChartDQA explicitly separates Basic chart types—defined in the paper as bar, line, and pie charts—from Advanced chart types. On ChartDQA, the reported ChartScope$_{\text{TinyLLaVA-3.1B@768}}$ achieves 47.1 on Basic and 38.3 on Advanced, compared with 27.7 and 22.1 for TinyChart, and 34.2 and 28.5 for ChartMoE (Fan et al., 18 Jul 2025). These are among the strongest results in the paper because they align with the framework’s stated goal of improving breadth across more specialized chart families.

The benchmark profile is not uniformly dominant across all tasks, which is an important corrective to overly broad interpretations. On ChartX, the TinyLLaVA-based ChartScope reaches 35.7, outperforming TinyChart’s 33.4. On PlotQA, which the paper treats as more diagnostic of underlying-data understanding because charts are unannotated, it reaches 35.2 versus TinyChart’s 32.6. On ChartQA, however, TinyChart remains slightly higher at 83.6 versus ChartScope’s 83.2; on Chart-to-Table, TinyChart also remains higher at 93.8 versus 93.3 (Fan et al., 18 Jul 2025).

The authors interpret this pattern as evidence that ChartScope’s main strengths lie in broader chart coverage and in settings where OCR shortcuts are weaker. A related nuance is that the paper repeatedly states coverage of 20 chart types, but its limitations section says that the current model supports only 18 chart types. This discrepancy is part of the framework’s current empirical profile rather than a resolved claim of open-ended chart universality (Fan et al., 18 Jul 2025).

6. Position within chart-understanding research and limitations

ChartScope belongs to a broader research landscape in which chart understanding is increasingly decomposed into structured perception and structured reasoning. Related systems emphasize different layers of this problem: CACHED addresses context-aware detection of chart elements and structural areas (Yan et al., 2023), CHARTER targets end-to-end chart-to-table extraction with heatmap-based chart primitive detection (Shtok et al., 2021), StructChart introduces Structured Triplet Representations (STR) and the SCRM evaluation metric (Xia et al., 2023), and OneChart produces Python-dict chart structures with a confidence mechanism derived from an auxiliary token (Chen et al., 2024). More recent work pushes further toward agentic or executable reasoning, as in ChartAgent with Tool-Integrated Reasoning and an Evidence Package (Wang et al., 16 Dec 2025), or toward multimodal reflection, as in ChartSketcher with sketch-mediated reasoning (Huang et al., 25 May 2025).

Against this backdrop, ChartScope’s emphasis differs from parser-centered systems and from chart-to-code systems. Its focus is not primarily chart element detection, explicit geometric tool orchestration, or code generation, but chart-data alignment inside an LVLM and broad chart comprehension through synthetic pretraining and Dual-Path supervision. This suggests a complementary role rather than a replacement for those systems.

The limitations stated in the paper are material. The synthetic pipeline can introduce incorrect raw data, imperfect data/code correspondence, and residual filtering failures. The breadth claim remains incomplete because, by the paper’s own account, the current model supports only 18 chart types even though the benchmark covers 20. The authors also note that some classical chart benchmarks may be saturated or annotation-heavy and may therefore reward OCR more than genuine chart-data understanding (Fan et al., 18 Jul 2025).

A common misconception would therefore be to treat ChartScope as a universal chart model or as a model that is uniformly best on all chart benchmarks. The reported evidence supports a narrower and more precise conclusion: ChartScope is a chart-specialized LVLM framework whose strongest contributions are chart-JSON alignment, Dual-Path supervision, and improved performance on broader and less annotation-dependent chart understanding settings (Fan et al., 18 Jul 2025).