Chart Synthesis Pipeline

• Chart Synthesis Pipeline is a systematic workflow that transforms structured inputs like tables, code, or SVG primitives into coherent, multimodal chart representations.
• It integrates metadata-driven, code-centric, and reverse-engineering approaches to generate visual marks, numerical data, and semantic layers, ensuring alignment across modalities.
• The pipeline employs execution, rendering, and semantic filtering to verify quality and consistency, supporting diverse tasks such as infographic generation and chart-to-code conversion.

A chart synthesis pipeline is a programmatic workflow that constructs chart images, chart specifications, semantic annotations, or reasoning supervision from structured intermediates such as tables, plotting code, metadata, SVG primitives, or real chart exemplars. Across recent work, charts are treated not as ordinary images but as layered objects that bind visual marks to numerical data, labels, layout, and analytical intent. One explicit formulation denotes chart metadata as $M=(T,X,Y,CT,D,E)$, where $T$ is the title, $X$ and $Y$ are axis labels, $CT$ is the chart type, $D=\{(x_i,y_i)\}_{i=1}^n$ is the underlying data, and $E$ denotes additional chart elements; semantic synthesis is then written as $f:(C,M)\rightarrow I$, with $I=(I^v,I^t,I^s)$ for visual-oriented, task-oriented, and statistics-oriented insights (Wu et al., 15 May 2025). Other pipelines instead make executable plotting code the central latent representation from which images, tables, summaries, masks, or reasoning traces are derived and verified (Kondic et al., 28 Mar 2026).

1. Representational foundations

A central theme in the literature is that chart synthesis is fundamentally a problem of intermediate representation. Metadata-centric pipelines begin from chart specifications, axis labels, data tables, legends, or generated JSON; code-centric pipelines begin from executable plotting programs; reverse-engineering pipelines reconstruct SVG or other structured primitives from a raster chart. This shifts synthesis away from unconstrained captioning toward explicit structure recovery or explicit program construction (Wu et al., 15 May 2025, Ji et al., 14 Apr 2025).

Code has emerged as an especially important substrate because it can be rewritten, executed, checked, and used to derive multiple aligned modalities. ChartNet states the motivating observation succinctly: unlike many natural images, charts are usually born from code, so executable plotting programs can serve as the scaffold tying together image, data, text, and reasoning (Kondic et al., 28 Mar 2026). ChartGen adopts the same principle in a two-stage formulation—image to code, then code to more code—while Socratic Chart approaches the inverse problem, converting raster charts into SVG primitives such as <rect>, <path>, and <text> so that an MLLM can reason over an explicit symbolic representation rather than over pixels alone (Kondic et al., 31 May 2025, Ji et al., 14 Apr 2025).

This representational shift also clarifies a common misconception. In these systems, “synthesis” rarely means image-only generation. It more often means generating a coherent multimodal tuple whose components are constrained to agree because they originate from the same table, plotting program, chart specification, or function chain. That distinction is explicit in pipelines for semantic insights, chart-to-code corpora, infographic generation, and reasoning-trace construction (Wu et al., 15 May 2025, Li et al., 24 May 2025).

2. Source representations and workflow families

Recent pipelines draw from several source representations. A metadata-driven retrieval pipeline starts from raw tables, automatically visualizes them, stores chart specifications, and synthesizes hierarchical semantic insights from those specifications (Wu et al., 15 May 2025). Code-guided dataset pipelines start from seed chart images, reconstruct executable plotting code, iteratively rewrite that code into new variants, and then execute the variants to obtain large chart-image/code corpora (Kondic et al., 31 May 2025, Kondic et al., 28 Mar 2026). Real-to-synthetic infographic pipelines begin from human-designed infographic charts, induce chart types, chart variations, and layout templates, and then fill those templates programmatically with new data, text, icons, palettes, and layouts (Li et al., 24 May 2025). Other workflows start from executable plotting programs directly and trace their rendering internals to synthesize instance-level masks or other structural supervision (Niu et al., 8 May 2026).

Scale varies substantially across these families. The semantic-insight pipeline for text-to-chart retrieval crawls 9,003 Kaggle datasets with a score of 10, retains 8,860 high-quality CSV tables after removing missing values, renders 69,166 charts through DeepEye and multi-library re-rendering, and synthesizes 207,498 semantic insights (Wu et al., 15 May 2025). ChartGen starts from 13K unique ChartQA seed images and expands them into 222.5K unique chart-image/code pairs covering 27 chart types and 11 plotting libraries through VLM-based reconstruction and iterative code augmentation (Kondic et al., 31 May 2025). ChartNet scales the same basic principle further, using 150,000 TinyChart seeds to generate 1.5 million samples spanning 24 chart types and 6 plotting libraries (Kondic et al., 28 Mar 2026). ChartMaster shifts the source distribution toward real scientific figures by downloading 30,071 arXiv papers, extracting 288,992 chart images, and retaining 242,479 executable Matplotlib triplets in ReChartPrompt-240K after code generation and execution filtering (Tan et al., 25 Aug 2025). SynChart pushes synthetic scale further still, reporting approximately 3.93M chart images and 75.42M annotations across nine chart types (Liu et al., 2024). For infographic charts, ChartGalaxy induces 75 chart types, 330 chart variations, and 68 layout templates from 19 chart-rich websites before rendering a million-scale D3.js-based corpus (Li et al., 24 May 2025).

These source choices determine what can later be synthesized. Table-first pipelines are strong at preserving exact values and axis semantics. Code-first pipelines are strong at multimodal alignment, execution-based verification, and style diversification. Real-exemplar pipelines are strong at realism in layout, typography, iconography, and long-tail design patterns. Program-tracing pipelines are strongest when pixel-accurate localization of chart elements is needed.

3. Semantic, structural, and instance-level synthesis

One major branch of chart synthesis enriches charts with semantic or structural annotations that are not readily available in raw corpora. In the retrieval setting, the pipeline in "Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insights" defines three complementary insight types: visual-oriented $I^v$, task-oriented $T$0, and statistics-oriented $T$1. Visual-oriented insights summarize patterns such as upward or downward trends; task-oriented insights describe practical analytical uses; statistics-oriented insights summarize the outputs of a 10-task statistical analysis module over metadata. The resulting texts are used only during training, but they reshape the learned embedding space so that user queries align with latent chart semantics rather than only with surface appearance (Wu et al., 15 May 2025).

A second branch synthesizes fine-grained structural supervision from rendering internals. ChartREG++ exploits executable Matplotlib programs, traces plotting API calls to Artist objects and container semantics, constructs a multi-granularity Artist-to-visual mapping, and recovers pixel-accurate instance masks by extracting the pixels contributed by each Artist. Because the mapping preserves API context, the pipeline distinguishes semantically different but visually similar elements, such as scatter markers versus line-series markers, and supports a primitive–part–composition hierarchy over 18 chart element types. Bounding boxes are then obtained by converting each instance mask to its tight enclosing box (Niu et al., 8 May 2026).

A related reverse-engineering strategy represents charts as SVG. Socratic Chart first classifies the chart type, then uses specialized agents for bars, lines, pie segments, text, legends, and axes to generate candidate SVG fragments, and finally uses an agent-critic to consolidate them into a single SVG representation. Bar charts are rendered as <rect> elements, line charts as <path> elements, pie charts as proportional arc paths, and text as <text> elements, with the SVG then supplied alongside the original chart to an MLLM for downstream reasoning (Ji et al., 14 Apr 2025).

For line-chart digitization, ChartZero shows a different structural emphasis. It trains exclusively on 100,000 synthetic matplotlib charts generated from 20 parameterized mathematical function families, obtains ground-truth instance masks directly from the renderer, and uses those masks to train a dual-headed U-Net with a Global Orthogonal Instance loss for thin-curve separation. Here the synthesized supervision is not semantic text or SVG code but renderer-derived geometric priors for curve extraction (Islam et al., 7 May 2026).

4. Reasoning-data construction

Another major branch of chart synthesis constructs question-answer data, rationales, or executable reasoning traces. "Synthesize Step-by-Step" formalizes the core idea as $T$2 and $T$3: instead of directly generating a question and answer, the model generates a question plus an executable rationale program, and the final answer is obtained by running that program with external tools. The method bootstraps rationale supervision from 28 manually authored templates over SVG-derived chart metadata, producing 356,606 template-generated QA pairs on ChartQA training images, and then uses a multimodal generator to expand this into large reasoning-heavy corpora (Li et al., 2024).

Chain of Functions pushes this programmatic idea further by making the reasoning trace itself the primary object. It uses JSON as the canonical chart representation, explores valid chains over 6 selection methods, 99 object functions, and 8 value functions, and only afterward translates the resulting function chains into rationales and questions with Qwen2.5-32B-Instruct. The resulting ChartCoF dataset contains a 1,451-question fine-grained evaluation set and a 50,329-question training set across 19 chart types, with chain lengths from 2 to 13 and explicit function-taxonomy labels for fine-grained evaluation (Li et al., 20 Mar 2025).

Other pipelines move the same principle closer to executable chart code. ChartReasoner first trains a chart-to-ECharts model on roughly 110K image-code pairs, then feeds inferred ECharts code and a question into a long-chain reasoning generator to construct the ChartThink dataset, retaining only samples whose generated answer exactly matches the benchmark answer; the retained reasoning corpus contains over 140K examples (Jia et al., 11 Jun 2025). ChartM$T$4 uses a retrieval-augmented template database spanning 9 major chart families and 62 minor chart types, generates executable data-generation and plotting code, synthesizes reasoning code for task-specific questions, and constructs a dataset with 38K charts and 142K QA pairs for training plus 2,871 evaluation samples (Xu et al., 4 Nov 2025).

Self-improving pipelines use synthesized charts and metadata as supervision without external chart labels. Chart-CoCa starts from unlabeled real charts, generates a chart description, converts the description into Matplotlib code, executes the code to obtain a synthetic chart $T$5 and extracted metadata $T$6, matches seed descriptive questions to $T$7 to produce aligned $T$8 triplets, and then trains a candidate-conditioned answer model that synthesizes a final answer from multiple sampled responses (Jiang et al., 16 Aug 2025). DuoChart instead combines 8,691 LLM-code charts and 7,997 arXiv charts into 105,096 QA pairs and about 9k reliable tool-use trajectories, so that crop tools and code-computation tools can be trained jointly with chart reasoning (Zhang et al., 3 Apr 2026).

5. Verification, filtering, and alignment

Quality control is a defining feature of mature chart synthesis pipelines. The strongest systems do not treat generation as sufficient; they treat execution, rendering, and answer verification as mandatory gates. ChartNet reports that 77% of generated code snippets execute successfully on average, then applies a VLM-based visual filter over eight defect categories. This filter classifies 36.5% of rendered images as having some visual error, and human validation on 3,157 random charts shows readability-affecting issues dropping from 14.9% before filtering to 5.9% after filtering (Kondic et al., 28 Mar 2026).

Execution filtering is the simplest and most common gate. ChartGen retains only executable Python plotting scripts and removes duplicate charts using identical image embeddings (Kondic et al., 31 May 2025). SynChart similarly executes generated plotting code and then uses human-in-the-loop correction of recurring code failures, improving Matplotlib chart generation success from 64.0% to 76.8% and yielding roughly 600K additional chart images (Liu et al., 2024). In semantic-text pipelines, validation can instead be human-judged rather than execution-based: the semantic-insight pipeline validates 50 captions with 100 crowd workers across completeness, consistency, diversity, and readability, with most ratings falling in agree or strongly agree categories (Wu et al., 15 May 2025).

CharTide makes verification itself the alignment objective. In its Inquiry-Driven RL framework, the semantic reward is

$T$9

the visual reward is

$X$0

and total reward is

$X$1

Here a frozen Inspector answers 10 atomic questions per chart, the modality-fusion dataset retains only cases with WebSSL similarity $X$2, and the RL set keeps only charts on which the Inspector reaches $X$3 on the source image (Zheng et al., 24 Apr 2026). This replaces holistic judging with fact-preservation checks under the principle of information invariance.

6. Uses, limitations, and design implications

Synthesized chart artifacts now support a broad range of downstream tasks. Hierarchical semantic insights are used to train text-to-chart retrieval systems; pixel-accurate instance masks support chart referring expression grounding; synthetic geometric priors enable zero-shot line-chart data extraction; and real-to-synthetic infographic corpora support infographic understanding, code-generation benchmarking, and example-based infographic generation (Wu et al., 15 May 2025, Niu et al., 8 May 2026, Islam et al., 7 May 2026, Li et al., 24 May 2025). In most of these settings, the synthesized artifacts are used primarily as training-time supervision, even when the deployed system later consumes only chart images and natural-language queries.

Limitations remain tightly coupled to the chosen intermediate representation. Metadata-driven insight synthesis depends on high-quality chart specifications and does not directly solve raster-only collections. Code-driven reasoning pipelines that rely on ECharts or Matplotlib inherit the coverage and semantics of those ecosystems. Large-scale code-guided generation still faces execution bottlenecks, skewed final distributions across chart types and plotting libraries, and residual annotation noise even after aggressive filtering. Line-specific pipelines remain confined to line charts and delegate legend matching or axis semantics to a VLM rather than learning those pieces from synthetic data alone (Jia et al., 11 Jun 2025, Kondic et al., 28 Mar 2026, Islam et al., 7 May 2026, Liu et al., 2024).

Taken together, these works suggest three durable design principles. First, synthesis is strongest when the canonical intermediate is executable or structurally explicit—JSON, plotting code, chart metadata, SVG, or typed function chains—rather than a free-form caption. Second, execution should be treated as verification, not merely as rendering. Third, supervision benefits from decomposition: visual form, numerical relation, and analytical purpose are learned more effectively when separated into distinct streams or distinct synthesized views than when collapsed into a single chart-text pair (Kondic et al., 31 May 2025, Li et al., 20 Mar 2025, Zheng et al., 24 Apr 2026). In that sense, the modern chart synthesis pipeline is less a single algorithm than an architectural pattern for constructing chart-aware multimodal supervision with controllable semantics and verifiable fidelity.