ChartCap: Dense Chart Captioning Framework

• ChartCap is a real-world chart captioning dataset that produces dense, type-specific, hallucination-free descriptions strictly based on visible chart elements.
• It employs a four-stage pipeline—filtering, classification, extraction, and caption synthesis with cycle consistency—to ensure high-quality outputs.
• Evaluation via the Visual Consistency Score and OCRScore demonstrates ChartCap’s precision in reconstructing charts from detailed captions.

Searching arXiv for recent and foundational papers on ChartCap and chart summarization. ChartCap is a large-scale, real-world chart captioning dataset and evaluation framework designed to train vision-LLMs to produce dense, type-specific, hallucination-free chart descriptions (Lim et al., 5 Aug 2025). In the broader literature, it sits within the chart-to-text or chart summarization problem, in which a system must read a chart image and generate a natural-language description grounded in visible structure, labels, values, and trends rather than in surrounding document context or latent world knowledge (Kantharaj et al., 2022). The ChartCap formulation emphasizes three linked requirements: captions should be comprehensive, they should follow chart-type-specific schemas, and they should exclude any information not discernible from the chart, using explicit “not specified” placeholders when necessary to suppress hallucination (Lim et al., 5 Aug 2025).

1. Origins and research context

Before the paper titled “ChartCap,” the field was organized primarily around chart summarization, chart-to-text generation, and accessibility-oriented figure captioning. “Chart-to-Text: A Large-Scale Benchmark for Chart Summarization” introduced a benchmark with two datasets totaling 44,096 charts and showed that even strong models often produced fluent summaries that still suffered from hallucinations, factual errors, and difficulty in correctly explaining complex patterns and trends in charts (Kantharaj et al., 2022). “LineCap” narrowed the scope to 3,528 real line-chart figures from scientific papers and prioritized high-level, accessibility-oriented descriptions of trends over low-level metadata, exposing the gap between synthetic and real chart captioning settings (Mahinpei et al., 2022).

Subsequent systems increasingly targeted chart-specific structure rather than treating charts as ordinary images. “ChartReader” unified chart derendering and comprehension without heuristic rules, using chart component detection and chart-aware embeddings to improve Chart-to-Text performance (Cheng et al., 2023). “ChartThinker” argued that chart summarization required stronger visual-language matching and reasoning, pairing a large chart-caption and chart-QA corpus with a contextual chain-of-thought and retrieval mechanism (Liu et al., 2024). “ChartAdapter” then framed chart summarization as a chart-aware large vision-LLM problem, inserting a lightweight transformer module between a chart image encoder and an LLM and training on a 190,618-sample ChartSumm dataset (Xu et al., 2024). In parallel, “ChartCards” treated captioning as one task within a broader chart-metadata generation framework, generating overview and analysis captions alongside tables, visualization code, and analytic signals (Wu et al., 21 May 2025).

This progression suggests a broad shift in the literature: from OCR-heavy or template-heavy pipelines toward models and datasets that explicitly encode chart structure, numerical grounding, and chart-type-specific semantics. ChartCap is a direct response to that trajectory, but its stated target is narrower and more specific: mitigating hallucination in dense chart captioning by ensuring that captions contain only visually verifiable content (Lim et al., 5 Aug 2025).

2. Dataset design and caption schema

ChartCap contains 565K real-world chart-image and caption pairs and was assembled from an original pool of 3.1M images from ArxivCap, ChartSumm-Knoema, ChartCheck, and ChartQA-train (Lim et al., 5 Aug 2025). Its supported chart types are line, bar, pie, histogram, scatter, area, bubble, choropleth map, and treemap. The train split contains 509K examples, and the test split contains 56K examples; a validation split is not specified (Lim et al., 5 Aug 2025).

The dataset defines dense captions as long-form descriptions that comprehensively cover the chart’s structural components and key insights, and it states that ChartCap’s captions are the longest among compared datasets, averaging 231.1 words (Lim et al., 5 Aug 2025). The captions are also type-specific. For line charts, the schema includes structural elements such as title, axes, legends, and labels, together with key insights such as retrieve values, extrema, comparisons, range, and trend. For histograms, the schema includes extrema, comparisons, and distribution characterization. For scatter and bubble charts, the relevant insights include correlations or trends, clusters, anomalies, ranges, and extrema. Choropleth maps add base map, color scale, geographic labels, data classes, and north arrow; treemaps add tiles, hierarchy levels, and color coding (Lim et al., 5 Aug 2025).

A defining design rule is the exclusion of extraneous information. Captions are intended to include only what is visible in the chart; if an element is absent, it is explicitly marked as “not specified.” The paper presents this not only as a representational choice but also as a negative instruction during training to reduce hallucinations (Lim et al., 5 Aug 2025). This is a notable departure from many earlier real-world datasets, where captions could incorporate surrounding article context. A common misconception is that dense captions necessarily amplify hallucination because they are longer; ChartCap argues the opposite, namely that density can coexist with visual faithfulness when the schema is constrained to discernible chart evidence and missing fields are rendered explicitly as “not specified” (Lim et al., 5 Aug 2025).

3. Four-stage pipeline and cycle-consistency verification

ChartCap’s captions are generated through a four-stage pipeline that uses only data discernible from the chart (Lim et al., 5 Aug 2025). Stage 1 filters non-chart images with InternVL2.5-8B. Starting from the 3.1M-image pool, this stage retains 1.2M single data-driven charts, and manual inspection confirms 100% precision, with no false positives retained (Lim et al., 5 Aug 2025).

Stage 2 performs chart type classification and title extraction using GPT-4o. The model classifies the image into one of the nine supported chart types and extracts the main title; if no title is present, the output is “not specified.” After this step, 577K chart images remain, and the reported manual evaluation accuracy is 99%, with minor errors attributed to ambiguous title placement (Lim et al., 5 Aug 2025).

Stage 3 extracts type-specific information. The task is split by granularity: GPT-4o handles coarse-grained tasks such as global trends, comparisons, legends, and labels, while Claude 3.5 Sonnet handles fine-grained tasks such as axes details, exact extrema, and precise numeric reads (Lim et al., 5 Aug 2025). The extracted elements are accumulated into a semi-structured representation with attributes including Type, Title, Axes, Legends, Labels, Retrieve Value, Find Extremum, Determine Range, Comparisons, Correlations or Trends, Distribution, Clusters, and Anomalies. If any attribute cannot be recovered, it is marked “not specified.” The reported manual evaluation accuracy for this stage is 94%, and the paper identifies logarithmic axes, scatter plots with weak or no correlation, and inset plots as known difficulties (Lim et al., 5 Aug 2025).

Stage 4 synthesizes the final caption with GPT-4o-mini, transforming the semi-structured attributes into sentence-level text without adding or omitting information and retaining explicit “not specified” mentions (Lim et al., 5 Aug 2025).

Quality control is organized around cycle consistency. The cycle is chart $\rightarrow$ caption $\rightarrow$ code $\rightarrow$ reconstructed chart, after which humans compare the reconstructed chart with the original (Lim et al., 5 Aug 2025). Captions are converted into Matplotlib code by Claude 3.5 Sonnet; execution errors trigger iterative debugging with the error message fed back to the model until the code runs. This process was applied to 68K samples and produced the final 56K high-quality test set. The paper reports that this verification method is approximately 24× faster than direct image-caption evaluation and achieves an appendix F1 of 94.7%, with precision 100.0% and recall 90.0% on 100 sampled pairs (Lim et al., 5 Aug 2025).

4. Visual Consistency Score and reference-free evaluation

ChartCap introduces the Visual Consistency Score, a reference-free metric intended to evaluate caption quality by regenerating a chart from the caption and comparing that reconstructed chart to the original image (Lim et al., 5 Aug 2025). For a caption $C_i$, an LLM generates Matplotlib code $G_i$, which produces a reconstructed image $\hat{I}_i$. A vision encoder then computes the cosine similarity between the original chart $I_i$ and $\hat{I}_i$, and the dataset-level score is

$$\text{VCS} \;=\; \frac{1}{N} \sum_{i=1}^{N} \text{Sim}(I_i, \hat{I}_i).$$

The implementation uses Claude 3.5 Sonnet for caption-to-code generation, an automatic debugging loop for code execution, and SigLIP2 encoders at 512 resolution in Large, So400M, and Base variants (Lim et al., 5 Aug 2025).

The paper also defines a text fidelity sub-metric, OCRScore, by extracting text sets $\mathcal{T}_i$ and $\rightarrow$0 from original and reconstructed charts with PaddleOCR and computing aggregated precision and recall:

$\rightarrow$1

$\rightarrow$2

A central claim of the metric is its independence from reference captions. Unlike BLEU, ROUGE, METEOR, or BERTScore, VCS compares reconstructed charts directly to original chart images rather than to a human-written textual target (Lim et al., 5 Aug 2025). On pairwise agreement with human judgments, the reported VCS results using SigLIP2-So400M are 79.33% for informativeness, 77.00% for accuracy, and 77.33% for fewer hallucinations; OCRScore reports 76.00%, 75.00%, and 74.00% on the same three axes, and both outperform the cited reference-based metrics (Lim et al., 5 Aug 2025). The paper also reports that caption-to-code distortions are rare: in a 100-pair analysis, captions from a ChartCap-trained model yielded 99% accurate code translation, whereas a low-density caption baseline yielded 96% fidelity, with 4% of cases showing placeholder values inserted by the LLM because the caption omitted crucial details (Lim et al., 5 Aug 2025).

This metric addresses a long-standing evaluation difficulty in chart captioning. Earlier work had already shown that overlap metrics and human judgments can diverge; for example, LineCap reported that BLEURT correlated better with human judgments than BLEU or CIDEr, but still imperfectly (Mahinpei et al., 2022). ChartCap’s proposal can therefore be read as an attempt to shift evaluation from reference overlap toward visual reconstructability.

5. Model training, benchmark results, and generalization

The paper evaluates ChartCap with both open-source and proprietary models, including InternVL2.5 in 78B, 38B, 26B, and 8B variants, Phi3.5-Vision-4B, ChartGemma-2B, ChartInstruct-Llama2-7B, and Claude 3.5 Sonnet (Lim et al., 5 Aug 2025). Supervised fine-tuning with LoRA is applied to InternVL2.5-8B and Phi3.5-Vision-4B on the 509K-example ChartCap train split, using the prompt “Please provide a detailed caption for the chart.” Selected reported hyperparameters include 1–2 epochs, batch size 12–192, AdamW, learning rate $\rightarrow$3, warmup ratio 0.05–0.1, cosine or constant scheduler, and LoRA rank 32 with alpha 32–64 and dropout 0–0.05 (Lim et al., 5 Aug 2025).

On the 56K human-verified ChartCap test set, Phi3.5-Vision-4BChartCap achieves SacreBLEU 23.82, ROUGE-L 0.3900, METEOR 0.4084, BERTScore 0.7427, VCS 0.8933 with SigLIP2 Large, 0.8829 with So400M, 0.9092 with Base, and OCRScore 0.5179 (Lim et al., 5 Aug 2025). InternVL2.5-8BChartCap reports SacreBLEU 19.47, ROUGE-L 0.3393, METEOR 0.3729, BERTScore 0.7238, VCS 0.8913, 0.8828, and 0.9068, and OCRScore 0.5089 (Lim et al., 5 Aug 2025). Selected baselines are substantially lower on reference-based and reconstruction-based metrics, including Claude 3.5 Sonnet with SacreBLEU 5.35, VCS Large 0.8834, and OCRScore 0.4868, and Phi3.5-Vision-4BOriginal with SacreBLEU 0.09, VCS Large 0.7782, and OCRScore 0.1438 (Lim et al., 5 Aug 2025).

Human evaluation on 100 samples reports that Phi3.5-Vision-4BChartCap is consistently preferred over Claude 3.5 Sonnet and the Phi3.5-Vision-4B base model for informativeness, accuracy, and fewer hallucinations (Lim et al., 5 Aug 2025). The generalization results are also striking. On VisText, the paper reports that InternVL2.5-8BChartCap and Phi3.5-Vision-4BChartCap surpass ground-truth captions in VCS, and that Phi3.5-Vision-4BChartCap is preferred over both ground-truth captions and Claude 3.5 Sonnet in human evaluation across all aspects (Lim et al., 5 Aug 2025). On the human-verified PEW subset of Chart-to-Text, ChartCap-trained models again surpass human-authored ground truth in VCS and strongly improve OCRScore (Lim et al., 5 Aug 2025). A plausible implication is that ChartCap’s caption schema rewards visually exhaustive reconstructions even when earlier human-authored summaries were concise, contextual, or selective.

The paper also reports several targeted analyses. ChartCap’s long captions correlate with higher VCS and OCRScore than other real-world datasets with average caption lengths of 34–62 words. Task allocation between GPT-4o and Claude 3.5 Sonnet is justified by differential accuracy: GPT-4o attains 96% on coarse-grained extraction versus 93% for Claude 3.5 Sonnet, while Claude 3.5 Sonnet attains 94% on fine-grained extraction versus 68% for GPT-4o (Lim et al., 5 Aug 2025). VCS is sensitive to specific structural errors; correcting misidentified maxima or minima yields a gain of 1.3%, axis hallucinations 6.1%, and omitted data series 4.7% (Lim et al., 5 Aug 2025).

6. Limitations and significance within chart understanding

ChartCap’s schema currently spans nine chart types and excludes multi-view or multi-chart composites, specialized scientific plots, and 3D charts (Lim et al., 5 Aug 2025). The source distribution may be biased toward specific domains, regions, and English-language content because it relies on publicly available chart sources such as arXiv and public data portals (Lim et al., 5 Aug 2025). The paper also notes style overfitting as a concern: a model may learn ChartCap’s stylistic conventions rather than a more general notion of chart description, which is why the authors recommend diverse evaluation with datasets such as VisText and Chart-to-Text as well as VCS (Lim et al., 5 Aug 2025). VCS itself has constraints, because it depends on LLM-based code generation and a single image-embedding similarity, which may underweight fine-grained aesthetic or stylistic differences (Lim et al., 5 Aug 2025).

Within the broader research landscape, ChartCap occupies a distinctive position. Earlier datasets and models variously emphasized benchmark scale, accessibility-oriented summaries, chart-aware architectures, retrieval-augmented reasoning, or multi-task metadata generation (Kantharaj et al., 2022); (Mahinpei et al., 2022); (Cheng et al., 2023); (Liu et al., 2024); (Xu et al., 2024); (Wu et al., 21 May 2025). ChartCap instead centers the dense caption itself as both a training target and an evaluable object whose correctness can be tested by regeneration. This suggests a different operational definition of faithfulness: not merely lexical overlap with a reference, but sufficient structural and numerical completeness for chart reconstruction.

The resulting conception of ChartCap is therefore twofold. In the narrow sense, it denotes the 565K-example dataset, its four-stage generation pipeline, cycle consistency-based human verification, and the Visual Consistency Score (Lim et al., 5 Aug 2025). In the broader sense, it names a mature phase of chart captioning research in which hallucination mitigation, chart-type-specific schemas, and reference-free, visually grounded evaluation become central design principles rather than auxiliary considerations.