Effective Chart Dataset (ECD): Robust Chart Analysis

• Effective Chart Dataset (ECD) is a benchmark dataset designed for robust, multimodal chart understanding with comprehensive image, code, data, and QA components.
• It is constructed via a staged, code-guided pipeline that includes seed acquisition, chart-to-code reconstruction, augmentation, and strict quality filtering to ensure high fidelity and diversity.
• ECD improves performance in applications such as chart reconstruction, data extraction, summarization, and visual question answering, as demonstrated by significant gains in benchmarks like ChartNet.

The Effective Chart Dataset (ECD) formalism denotes a benchmark or training dataset constructed with the explicit aim of facilitating robust and generalizable chart understanding by machine learning models, particularly vision-LLMs (VLMs) and multimodal LLMs (MLLMs). The ECD paradigm is exemplified by datasets that systematically cover a broad spectrum of chart types, plotting libraries, scientific domains, and modalities (image, code, raw data, text, QA), and that adhere to rigorous protocols for algorithmic construction, quality filtering, cross-modal alignment, and formal evaluation. Several prominent instances—ChartGen, ChartNet, the ECD dataset for MLLM training, and the ChartInsighter benchmark—serve as reference implementations, each operationalizing the principles of ECD according to the requirements of their respective chart-focused downstream tasks (Kondic et al., 31 May 2025, Kondic et al., 28 Mar 2026, Yang et al., 8 Aug 2025, Wang et al., 16 Jan 2025).

1. ECD Construction Pipelines: Algorithmic Foundations

An ECD is typically constructed via a code-guided, staged pipeline with strong automation and modularity. The canonical workflow consists of:

1. Seed Acquisition: Selection of real or synthetic chart images as initial seeds, often from curated sources (e.g., TinyChart in ChartNet).
2. Chart-to-Code Reconstruction: Employing a vision-LLM to reconstruct the plotting script for each seed image, ensuring initial semantic grounding through execution validation (e.g., success rate of executable code as acceptance criterion).
3. Code-Guided Augmentation: Iterative transformation of chart code via a LLM, introducing controlled chart-type changes, style variations, plotting library conversions, and data/label perturbations. Each candidate code is executed and rendered; only successful, non-redundant outputs are retained (Kondic et al., 31 May 2025, Kondic et al., 28 Mar 2026).
4. Multimodal Attribute Generation: For each (image, code) pair, extraction and alignment of chart data (CSV), textual summaries, and question-answering items via LLM/QA pipelines.
5. Quality Filtering: Automated vetting (e.g., VLM-based defect detection, confidence scoring), discarding samples with visual, semantic, or rendering anomalies.
6. Subsampling and Stratification for Evaluation: Held-out splits or test sets are generated by stratified sampling across chart types and plotting libraries to ensure statistical coverage (Kondic et al., 31 May 2025, Yang et al., 8 Aug 2025, Kondic et al., 28 Mar 2026).

Each step is often formalized in pseudocode, with the augmentation step modeled as a random walk over a set of code transformations $\Gamma$: for each $t$, transformation $\tau_t$ is sampled uniformly from $|\Gamma|$, and applied as $C_t = f_{\gamma}(C_{t-1})$, with dropout for failed executions or duplicates. Final dataset size scales as $N \approx |D_0| \cdot (1 + (1-\epsilon) T)$, where $D_0$ is the set of successfully reconstructed seeds, $T$ the average number of augmentations per seed, and $\epsilon$ the duplicate/dropout rate (Kondic et al., 31 May 2025).

2. Modality, Coverage, and Composition

Effective Chart Datasets manifest comprehensive multimodal support, with core records containing:

• Plotting code (Python or JSON-based spec)
• Image of the rendered chart
• Structured data table (CSV or DataFrame)
• Natural language summary or caption
• One or more QA pairs, frequently with explicit chain-of-thought rationales or modality-bridged reasoning
• Optional attributes: document structure tags (DocTags), multi-subplot layouts, and domain-specific metadata

Diversity is systematically induced by sampling key parameters:

• Chart types (e.g., ChartGen: $T=27$, ChartNet: $t$0)
• Plotting libraries (e.g., ChartGen: $t$1, ChartNet: $t$2)
• Scientific themes or domains (25+ in ECD (Yang et al., 8 Aug 2025))
• Stylistic augmentation (palette, markers, labels, annotations, font attributes)
• Subplot configurations (ECD: over 252 combinations)

Quantitative proxies for visual diversity include color Shannon entropy (e.g., $t$3 vs.\ $t$4 in ChartGen), semantic embedding spread, and pixel-level entropy (e.g., $t$5 in diversified ECD vs.\ $t$6 without diversification) (Kondic et al., 31 May 2025, Yang et al., 8 Aug 2025).

3. Data Curation and Quality Filtering

Robust curation protocols are central to ECD construction. Quality control operates on two axes:

• Automated Filtering: Visual output is screened using VLMs or LLMs for eight or more error classes (missing data, labeling, legend, overlap, semantic mismatch, clarity, scale, anomalous rendering). ChartNet reports discarding $t$7 of renders after defect detection; human-verified error rates drop to $t$8 post-filtering (Kondic et al., 28 Mar 2026).
• LLM-Based Rating: For charts with high visual complexity, LLMs (e.g., GPT-4o in ECD) rate samples for visual fidelity and semantic accuracy; images with $t$9—an average of visual and semantic scores—below dataset mean $\tau_t$0 are discarded, typically removing $\tau_t$1 of candidates (Yang et al., 8 Aug 2025).
• QA Filtering: Only high-confidence ($\tau_t$2) QA items generated per chart are retained, eliminating $\tau_t$3 of low-confidence pairs in ECD (Yang et al., 8 Aug 2025).

4. Formal Evaluation Subsets and Metrics

Held-out evaluation is a defining feature, designed for model benchmarking and ablation studies. Construction follows stratified uniform sampling across chart types and libraries, e.g., ChartGen's $\tau_t$4 split with at least $\tau_t$5 per chart type (for $\tau_t$6, $\tau_t$7) (Kondic et al., 31 May 2025).

Core evaluation metrics include:

• Execution Rate ($\tau_t$8): Fraction of predicted scripts that run without error
• Data Fidelity ($\tau_t$9): Pointwise matching of predicted chart data to ground truth
• Semantic/Style Consistency ($|\Gamma|$0): Categorical and stylistic alignment, often LLM-judged on a $|\Gamma|$1 scale
• Image Similarity ($|\Gamma|$2): Visual overlap between rendered images (ground truth vs. prediction)
• For QA components: fuzzy answer match for reasoning, chain-of-thought alignment, and model output robustness across descriptive/reasoning tasks

Example: ChartGen’s VLM benchmark results (best model, Granite-Vision-3.1) achieved $|\Gamma|$3 (data fidelity) and $|\Gamma|$4 (image similarity), indicating substantial headroom relative to oracle performance (Kondic et al., 31 May 2025).

5. Applications and Empirical Gains

Fine-tuning VLMs and MLLMs on ECDs substantially improves performance on chart reconstruction, data extraction, summarization, and visual question answering tasks across both synthetic and real-world benchmarks (Kondic et al., 31 May 2025, Kondic et al., 28 Mar 2026, Yang et al., 8 Aug 2025). In ChartNet, post-training improvements as measured on the core suite included:

Task	Metric	Base (%)	+ChartNet (%)	Absolute Gain
Reconstruction	Exec. rate	63.4	90.4	+27.0
Data Fidelity	Code-D	60.7	72.8	+12.1
Struct. Overlap	Code-S	67.0	90.0	+23.0
Img. Similarity	Img.	77.2	92.8	+15.6
Data Extraction	CSV acc.	53.8	70.3	+16.5
Summarization	Holistic	64.0	83.9	+19.9
QA w/ Reasoning	Fuzzy acc.	59.9	65.0	+5.1

This reflects the central role of ECDs in enabling transferable chart reasoning and code generation capabilities (Kondic et al., 28 Mar 2026).

6. Annotated Benchmarks and Diagnosis (Time-Series Summarization)

Variant ECDs also exist as specialized benchmarks for diagnostic purposes. ChartInsighter’s ECD comprises 75 time-series charts, each with a manually written "gold" summary and three model-generated summaries, with sentence-level hallucination annotations for 10 error categories (e.g., extremum, value, trend direction, omission) (Wang et al., 16 Jan 2025). Human annotation follows a detailed protocol: sentences are labeled for summary level (L1–L3, per Lundgard & Satyanarayan) and any hallucination tags. Baseline hallucination rates per sentence: VL2NL $|\Gamma|$5, GPT-4 $|\Gamma|$6, ChartInsighter $|\Gamma|$7; semantic richness of summaries is also quantified (mean richness for ChartInsighter: $|\Gamma|$8), supporting granular progress tracking for summarization algorithms.

7. General Design Principles and Open Challenges

The collective findings from ChartGen, ChartNet, ECD, and ChartInsighter articulate the following guiding principles for Effective Chart Datasets:

1. Multimodal Grounding: Use of code as the central linking modality, aligning image, data, text, and QA.
2. Automated, Modular Synthesis: Staged and scalable pipelines leveraging VLM/LLM orchestration and code- or prompt-level diversification.
3. Rigorous Filtering and De-duplication: Systematic application of both automated and human quality controls, with explicit filtering criteria.
4. Balanced and Diverse Coverage: Stratified sampling to ensure statistical balance across chart types, domains, and style variants.
5. Formal Benchmarks with Public Evaluation: Construction of stratified, held-out evaluation subsets using standardized metrics and public release of code, prompts, and annotations for transparency and reproducibility (Kondic et al., 31 May 2025, Yang et al., 8 Aug 2025, Kondic et al., 28 Mar 2026, Wang et al., 16 Jan 2025).

Noted limitations include the computational and financial cost of large-scale LLM/VLM runs (e.g., GPT-4o usage in ECD), the need for improved domain-specific perceptual metrics, and open questions about optimal QA-type mixtures and scaling behavior beyond current dataset sizes.

A plausible implication is that as domain-specialized vision encoders and semantics-aware metrics are developed, future ECDs may further enhance both fidelity and benchmarking granularity, enabling deeper scientific chart reasoning.