Chart-to-Code Generation

• Chart-to-code generation is an automated process that translates visual or textual chart inputs into precise, executable code.
• It integrates multi-stage pipelines, reinforcement learning, and iterative refinement to achieve lossless representation and high fidelity.
• The technique enhances data extraction, reproducibility, and accessibility across domains like education, business intelligence, and scientific research.

Chart-to-code generation is the computational process by which visual or descriptive representations of charts—such as images, infographics, flowcharts, or analytical texts—are automatically translated into executable code capable of reconstructing the original chart with high fidelity. This process is foundational for lossless chart understanding, automated data extraction, visualization reproducibility, and advanced multimodal reasoning in LLMs and vision-LLMs (VLMs). The landscape of chart-to-code research encompasses a wide variety of approaches, including multi-stage pipelines, reinforcement learning, knowledge graph extraction, code-based preference optimization, and agent-based iterative refinement, with applications spanning scientific publishing, education, business intelligence, accessibility technologies, and AI-driven report generation.

1. Problem Formulation and Core Challenges

Chart-to-code generation requires models to integrate several nontrivial capacities:

• Visual Understanding: Decoding chart type, axes, legends, layout, colors, data series, annotations, and complex semantics from visual or textual input.
• Code Translation: Mapping the cognitively parsed structure into executable code (e.g., matplotlib, ECharts, D3.js, Plotly) that, when run, produces a chart visually and semantically indistinguishable from the source.
• Lossless Representation: Ensuring that all critical details (data, style, layout, interactivity) are preserved.

The process is often formalized as: $C = f(X, I)$ where $X$ denotes the chart input (image or text), $I$ is any accompanying instruction, and $f$ is the model producing code $C$.

Principal challenges include:

• Information loss due to incomplete extraction or hallucination, particularly for dense or complex chart images.
• Low code executability rates, often due to syntax or logic errors in generated code.
• Limited diversity and size of training datasets, resulting in brittle generalization, especially for out-of-distribution chart types.
• Handling of semantic ambiguity, incomplete data, uncertainty, and subjective intent present in text-based or infographic sources.

2. Key Methodologies and Pipelines

Numerous frameworks address chart-to-code generation, distinguished by their structuring of the problem and level of automation:

Multi-Stage and Modular Pipelines

Text2Chart uses a three-stage sequence of axis entity recognition (BERT+BiLSTM for token labeling), entity mapping (Random Forest for $x$–$y$ association), and chart type prediction (fastText+LSTM), culminating in code or chart specification generation (2104.04584).

ChartKG leverages computer vision (ResNet50 for chart type, YOLOv5 for element detection, Tesseract for OCR) to construct a comprehensive knowledge graph—encoding visual elements, data variables, and their relationships—for downstream code or information extraction tasks (2410.09761).

ChartIR separates visual understanding from code translation, employing structured natural language instructions ("description" for all visual facets; "difference" for correction guidance) and iterative refinement, thus enabling systematic correction of visual-code mismatches (2506.14837).

Iterative, Preference-Driven, and Reinforcement Learning Approaches

Chart2Code introduces iterative dual preference learning, generating code variants along specified chart dimensions (type, data, layout, color, text, style), ranking outputs via dual reward signals—one for code structure (F1) and another for visual fidelity (multi-aspect binary scoring)—and optimizing model updates for robust, lossless code generation (2504.02906).

Text2Chart31 achieves RL-based instruction tuning using automated preference and alignment rewards: chart description–code cycles are regularized via BertScore-based alignment, while code preference models are trained using Proximal Policy Optimization on supervised vs. predicted code pairs (2410.04064).

ChartReasoner takes a code-driven approach: a pretrained Chart2Code model generates ECharts code ("symbolic transport"), which is then used as input for an LLM to create reasoning trajectories and answers, with training driven by supervised fine-tuning and Group Relative Policy Optimization reward signals (2506.10116).

ChartCoder employs a code LLM backbone (DeepSeek Coder 6.7B) and a "Snippet-of-Thought" process, decomposing chart-to-code tasks into four explicit structured steps, teaching the LLM to attend to each chart attribute before producing the final executable code (2501.06598).

Multi-Agent and Feedback-Driven Systems

METAL decomposes chart-to-code into an iterative collaboration among generation, visual critique, code critique, and revision agents (VLM-based). At each refinement iteration, code and rendered charts are evaluated on multi-level metrics (e.g., text, color, layout F1), with agentic decision-making driving focused improvement (2502.17651).

PlotEdit utilizes a multi-agent system for PDF chart editing. Its Chart2Code agent (driven by GPT-4V) receives structured data and style hints from upstream extraction agents, then iteratively refines code using multimodal feedback (AST validation, sandbox execution, visual similarity via SSIM), closing the loop between data, style, and code (2501.11233).

3. Datasets and Evaluation Benchmarks

The field has witnessed rapid growth in dataset diversity and realism, supporting robust chart-to-code training and evaluation.

Synthetic and Large-Scale Corpora

• GenPlot programmatically generates billions of varied plain charts (bar, line, dot, scatter) with synthetic data, offering controllable metadata and label diversity for massive-scale denoising and generalization (2306.11699).
• SynChart produces ~4M charts with dense code, data, descriptions, and QA pairs; code is generated for multiple plotting libraries, with human-in-the-loop refinement enhancing code executability (2409.16517).
• ChartCards and MetaChart provide 85K+ charts along with code, tables, element-level metadata, and rich captions, supporting a wide variety of chart-understanding and code-generation tasks (2505.15046).
• ChartGalaxy targets infographics, offering >1M real and synthetic charts with paired data, 75 chart types, 330 layout/style variations, and low-level code generation benchmarks for D3.js (2505.18668).
• Text2Chart31 and ReachQA stress diversity (31 and 32 chart types) including 3D, gridded, and volumetric plots, with associated code and reasoning steps (2410.04064, 2410.18798).

Benchmarks for Code Generation and Fidelity

• ChartMimic comprises 1,000 human-curated image–instruction–code triplets (22 chart types, 191 subcategories) with multi-level (GPT-4V and structural F1) evaluation, directly measuring chart-to-code fidelity in both direct and customized mimic tasks (2406.09961).
• Flow2Code benchmarks chart-to-code generation from flowcharts (code/UML/pseudocode) across 15 programming languages and numerous code segments, exposing limits in code generalization from complex diagrams (2506.02073).

Metrics

Evaluation includes:

• Execution Rate: Fraction of generated scripts that run without errors (2501.06598).
• Structural/Low-Level F1: Text, type, color, layout F1 scores via code tracing (2406.09961).
• High-Level Similarity: GPT-4V/4o image-to-image scoring (0–100) (2406.09961, 2505.18668, 2506.14837).
• Dual Reward Signals: Binary visual aspect scoring and code structure comparison (2504.02906).
• User and Expert Evaluations: For real-world utility and error analysis, especially on complex or expressive charts (2410.14331, 2505.18668).

4. Applications and Broader Implications

The rapid advances and increasing robustness of chart-to-code generation methods have catalyzed a spectrum of applications:

Domain	Application Example
Scientific Publishing,	Recovering data/code from figures in academic papers, assuring reproducibility
Business Intelligence	Automating chart generation for dashboards and reporting, reverse engineering visualizations
Accessibility	Converting image-only figures (e.g., in PDFs) into editable, accessible code or tables
Education & E-Learning	Automated chart authoring and grading in courses, interactive chart analysis tools
Data Journalism	Fact-checking and extracting underlying data from infographics
Software Documentation	Embedding executable charts from documentation images or data

Additionally, methodologies that represent chart information as executable code or knowledge graphs (e.g., ChartKG, ChartReasoner, ChartCards) are being adopted to power chart question answering, cross-modal retrieval, and explanation tasks—demonstrating the critical link between code-driven generation and fine-grained chart reasoning.

5. State-of-the-Art Models and Research Frontiers

Models such as ChartCoder, ChartLlama, SynChart, ChartReasoner, and ChartIR achieve superior chart-to-code fidelity, often approaching or in some metrics surpassing proprietary LLMs like GPT-4o, especially in open-source scenarios (2501.06598, 2311.16483, 2409.16517, 2506.10116, 2506.14837). Key developments driving these advances include:

• Dedicated code LLM backbones for improved executability (ChartCoder, ChartReasoner).
• Large-scale, diverse, human-validated datasets (ChartGalaxy, SynChart, MetaChart).
• Iterative refinement and explicit error correction using structured instructions or multi-agent critique (ChartIR, METAL, PlotEdit).
• Dual-modality reward optimization and structured variant comparison (Chart2Code).

Research continues on:

• Scaling to more complex chart types, layouts, and real-world visual noise.
• Improving preference/reward modeling with meta-feedback and human-in-the-loop signals.
• Broadening cross-modal generalization—allowing chart-to-code methods to benefit broader multimodal reasoning.

6. Open Challenges and Future Directions

Although substantial progress has been made, outstanding issues include:

• Handling code and visual hallucination, especially for uncommon chart types or ambiguous input.
• Robustness to noisy, incomplete, and nonstandard input (e.g., hand-drawn or scanned charts).
• Generalizing across languages and visualization paradigms (beyond Python/matplotlib).
• Integration of user-in-the-loop corrections and interactive refinement.
• Balancing computation cost with iterative and multi-agent refinement strategies.

Future research directions call for deeper reward modeling, curriculum- and feedback-based learning, expansion to richer visualization grammars, and tighter integration with real-world design and accessibility workflows (2505.18668, 2506.14837).

7. Summary Table: Representative Frameworks, Datasets, and Advances

Framework/Dataset	Notable Features	Performance/Advances
ChartCoder & Chart2Code-160k (2501.06598)	Code LLM backbone, 160k executable pairs, SoT reasoning	91%+ code exec. rate, SoTA open
ChartIR (2506.14837)	Structured description/difference, iterative prompt-based refinement	Outperforms direct/METAL on hard sets
ChartMimic (2406.09961)	1,000 triplets, multi-level eval, 22 categories	Proprietary/open models benchmarked
ChartGalaxy (2505.18668)	>1M infographics, 75 types, code, low/high-level eval	Sets SOTA code-gen fidelity for infographics
PlotEdit (2501.11233)	Multi-agent code/data/style extraction & feedback loops	State-of-the-art chart recovery/editing
SynChart (2409.16517)	4M charts, 75M annotations, multi-engine code	84.6% ChartQA-Avg, near GPT-4O
ChartKG (2410.09761)	KG mapping of visual elements & semantics	Boosts VQA and semantic retrieval
Flow2Code (2506.02073)	16.8k flowcharts, 15 languages, code-gen benchmark	Gemini-2.0 leads, SFT crucial
Text2Chart31 (2410.04064)	31 types, RL-instruction-tuning, 3D/volumetric/gridded	SFT+RL small models surpass GPT-4o

---

Chart-to-code generation constitutes a rapidly evolving field, with advances driven by synthetic and real-world datasets, model scaling and specialization, explicit preference- and feedback-based training, and cross-modal generalization strategies. Its integration of visual, linguistic, and programmatic reasoning is advancing the broader goal of machine comprehension and automation in data-rich scientific, business, and design domains.