Charts-of-Thought: Structured Chart Reasoning

• Charts-of-Thought is a structured method that decomposes chart reasoning into clear steps using explicit intermediate representations such as tables, graphs, and diagrams.
• It leverages techniques like data extraction, table creation, and DAG-based processing to facilitate precise numerical and logical analysis of visual data.
• Empirical results show significant improvements in visualization literacy and reasoning performance, suggesting practical applications in multimodal chart analysis.

Searching arXiv for the cited papers to ground the article. Charts-of-Thought denotes a line of work in which chart interpretation and related multimodal reasoning are organized through explicit intermediate structures rather than a single unconstrained answer pass. In the narrow sense, “Charts-of-Thought” is a structured prompting method for visualization literacy that requires data extraction, table creation, sorting, verification, and question analysis before answering (Das et al., 6 Aug 2025). In a broader sense, closely related work represents chart reasoning as question-specific directed acyclic graphs, executable code, tabular schemas, conceptual diagrams, or node-link semantic canvases (Zhang et al., 2024, Tang et al., 11 Sep 2025, Sun et al., 4 Jan 2025, Borazjanizadeh et al., 14 Mar 2025, Li et al., 3 Mar 2026). Taken together, these papers suggest a common program: make reasoning over charts and chart-like structures explicit, inspectable, and, where possible, verifiable.

1. Conceptual scope and representational family

A central claim shared across this literature is that chart reasoning is not adequately modeled as direct visual question answering. Chart Question Answering is described as strongly context dependent and requiring complex logical and numerical reasoning (Zhang et al., 2024). Visualization literacy work makes a closely related argument: standard prompts often ask a model to answer directly from an image, but do not guide it through the recognition, extraction, verification, and analysis process that humans use when reading charts (Das et al., 6 Aug 2025).

The representational forms differ, but their role is similar: each externalizes or structures intermediate reasoning so that the model does not rely entirely on a single latent pass. Related work spans chart-specific DAG execution graphs, table-based reasoning schemas, code-based symbolic reasoning, conceptual diagrams for planning, and semantic canvases that transform spoken thought into node-link structure (Zhang et al., 2024, Sun et al., 4 Jan 2025, Tang et al., 11 Sep 2025, Borazjanizadeh et al., 14 Mar 2025, Li et al., 3 Mar 2026).

Representation	Function	Exemplars
Directed acyclic graph	Encode operator dependencies or reasoning roles	GoT-CQA; DoT
Table or verified outline	Organize values, constraints, or extracted content	Charts-of-Thought; Table as Thought
Executable code	Symbolize chart data and compute answers	Code-as-Thought; ChartVerse; Chart-R1
Diagram or canvas	Externalize states, concepts, or spoken content	Visualizing Thought; Orality

A common misconception is that Charts-of-Thought is merely chart-specific chain-of-thought. The cited work distinguishes these paradigms explicitly. GoT-CQA contrasts linear Chain-of-Thought with a structured DAG whose nodes correspond to reasoning operators (Zhang et al., 2024). “On the Diagram of Thought” pushes this further by treating reasoning as an internally constructed DAG navigated by a single auto-regressive LLM, rather than an externally orchestrated search (Zhang et al., 2024).

2. Formal structures: graphs, tables, diagrams, and canvases

The most explicit chart-domain formalization is GoT-CQA. For a question $Q$, the reasoning process is represented as a DAG

$$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$

with $$\text{type}(o_i)\in\{\text{Loc},\text{Num},\text{Log}\}$$ and directed edges encoding sub-step dependencies (Zhang et al., 2024). This is a compositional view of chart reasoning: localization identifies relevant marks or legend items, numerical operators read or infer values, and logical operators perform comparison, summation, subtraction, maximum, or average-based comparisons.

Table-based work shifts the structure from inter-step dependencies to the internal content of each step. “Table as Thought” defines a schema $S$ for a query $Q$, initializes a reasoning table $T$, and iteratively updates it until $Sufficient(T,Q)$ holds (Sun et al., 4 Jan 2025). Rows are sequential thought steps and columns are task-relevant fields such as constraints, variables, intermediate values, or final decisions. The paper’s claim is not merely that thoughts should be sequential, but that the content of each thought should be explicitly typed and verifiable.

Diagram-oriented work generalizes the same idea beyond charts. “On the Diagram of Thought” formalizes reasoning as a categorical diagram

$D:\mathcal{J}\rightarrow\mathcal{E}$

and interprets the final synthesis step as the colimit $\varinjlim D$ (Zhang et al., 2024). The framework uses learned role tokens such as <proposer>, <critic>, and <summarizer> to let a single model propose, critique, refine, and synthesize within an internal DAG. The paper claims logical consistency and soundness, but also notes, in effect, that the guarantee is better understood as formal grounding and conceptual justification than as a fully formal end-to-end theorem.

Other work externalizes thought into explicit visual artifacts. “Visualizing Thought” represents each planning state as

$s_d = \big(T(s_d), D(s_d), A_{0:d}\big),$

combining a textual state description, a conceptual diagram, and the action history (Borazjanizadeh et al., 14 Mar 2025). Orality uses a node-link “Semantic Content Space” in which topic nodes and content nodes form a manipulable externalized mental model from spoken input (Li et al., 3 Mar 2026). These are not chart QA systems, but they reinforce the same representational principle.

3. Chart-specific reasoning pipelines

GoT-CQA is the clearest chart-domain execution architecture. The chart is first converted into a sequence representation $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$0 using Donut, followed by self-data reasoning: a fully connected preprocessing layer maps $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$1, multiple self-attention layers mine relationships among chart elements, and the output is the feature matrix $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$2 (Zhang et al., 2024). Each GoT node is then executed by an operator-specific block: $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$3 If a node has no incoming edges, it starts from the global chart representation; otherwise it consumes outputs from prerequisite nodes. Textual guidance $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$4 is encoded by a LLM such as BERT, and the paper reports that the best configuration is SELF-CROSS: self-attention refines internal structure and cross-attention couples textual guidance with chart data.

The named “Charts-of-Thought” prompting method implements a lighter-weight but related pipeline. The model is instructed to extract all visible numerical values, identify values from both axes when applicable, create a markdown table with appropriate headers and units, sort the data, compare the table against the chart, correct mismatches, confirm the corrected table is final, and then answer using only the verified data (Das et al., 6 Aug 2025). The prompt is explicit: “Using ONLY the verified data in your table, compare EACH value individually with the reference value…”. The method is therefore a constrained chart-reading protocol, not a free-form answer prompt.

These two systems differ in where structure is imposed. GoT-CQA hardwires structure into a learned execution graph over Loc, Num, and Log operators (Zhang et al., 2024). Charts-of-Thought imposes structure at prompt time through extraction, verification, and tabulation (Das et al., 6 Aug 2025). A plausible implication is that the former targets compositional execution inside a chart QA model, whereas the latter targets inference-time discipline in general multimodal LLMs.

4. Programmatic and adaptive chart thought

Programmatic representations introduce a stronger form of verifiability. “Visual Programmability” proposes Code-as-Thought, in which a model extracts chart data into a structured form such as a pandas.DataFrame named chart_data and computes the answer programmatically (Tang et al., 11 Sep 2025). Its key factorization is

$$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$5

where $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$6. The central claim is that chart reasoning should be adaptive: some chart-question pairs have high Visual Programmability and benefit from code, while others are better handled by direct visual reasoning.

The reinforcement-learning setup makes that distinction explicit. The model is trained with GRPO and a weighted reward

$$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$7

with final weights $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$8, $$\mathcal{G}=\{\mathcal{O},\mathcal{E}\},\quad \mathcal{O}=\{o_1,\dots,o_n\},\quad o_i=(\widetilde{o}_i,\text{type}(o_i)),$$9, $$\text{type}(o_i)\in\{\text{Loc},\text{Num},\text{Log}\}$$0, and $$\text{type}(o_i)\in\{\text{Loc},\text{Num},\text{Log}\}$$1 (Tang et al., 11 Sep 2025). The data-accuracy reward applies only to the <[CODE](https://www.emergentmind.com/topics/confident-ordinary-differential-editing-code)> path and scores column completeness, row completeness, and value accuracy; the decision reward teaches when code is appropriate at all.

ChartVerse approaches the same problem from the data side. It introduces Rollout Posterior Entropy as a chart complexity metric, uses $$\text{type}(o_i)\in\{\text{Loc},\text{Num},\text{Log}\}$$2 to define “hard” charts, synthesizes code-grounded charts from scratch, and then performs truth-anchored inverse QA synthesis by extracting deterministic answers from source code before generating questions (Liu et al., 20 Jan 2026). The resulting datasets, ChartVerse-SFT-600K and ChartVerse-RL-40K, are designed to be both complex and answer-reliable.

Chart-R1 similarly treats chart reasoning as a reasoning-first training problem. It generates code-first chart reasoning data grounded in real tables from arXiv papers, uses Chart-COT for step-by-step supervision, and then applies Chart-RFT with GRPO and a numerically sensitive reward. For numerical answers, the reward uses a ±5% relative error tolerance; for string answers, it uses edit distance (Chen et al., 21 Jul 2025). Here the “thought” is not a graph or table alone, but a trainable chain of decomposed reasoning steps.

5. Empirical results across chart and diagram reasoning

GoT-CQA reports strong chart QA results on ChartQA and PlotQA-D. On ChartQA, it achieves 47.1 on human-written questions, 87.9 on augmented questions, and 67.5 overall (Zhang et al., 2024). On PlotQA-D1, it reports 98.4 on S, 97.8 on D, 86.5 on R, and 92.8 overall; on PlotQA-D2, 98.2 on S, 88.8 on D, 72.6 on R, and 78.3 overall. The paper emphasizes that performance drops from structural to retrieval to reasoning across methods, but that GoT-CQA remains especially competitive on reasoning questions.

The named Charts-of-Thought prompting method improves visualization literacy across all tested models on VLAT. On the modified VLAT, GPT-4.5 improves from 23.50 to 35.67 VLAT score, Gemini-2.0 from 37.72 to 42.78, and Claude-3.7 from 39.89 to 49.44; the reported relative raw-score improvements are 21.8% for GPT-4.5-preview, 9.4% for Gemini-2.0-pro, and 13.5% for Claude-3.7-sonnet (Das et al., 6 Aug 2025). On the original VLAT, Claude-3.7-sonnet reaches a VLAT score of 50.17, compared with the human baseline of 28.82.

Adaptive programmatic reasoning also shows a strong benchmark pattern. The Visual Programmability model achieves 62.8% average accuracy across ChartX, ChartBench, ChartQA, and CharXiv (Tang et al., 11 Sep 2025). Its code usage is about 76.0% on ChartX, 66.6% on ChartBench, 98.3% on ChartQA, and 10.1% on CharXiv, which is presented as the expected behavior under the Visual Programmability hypothesis. The appendix also shows the brittleness of fixed code-only reasoning: a CaT specialist reaches 71.6% on ChartX but collapses to 18.4% on CharXiv.

On the data-scaling side, ChartVerse-8B reaches an average of 64.1 on chart benchmarks, exceeding its teacher Qwen3-VL-30B-A3B-Thinking at 62.9 and approaching Qwen3-VL-32B-Thinking at 67.0 (Liu et al., 20 Jan 2026). Chart-R1-7B reports 91.04 on ChartQA, 46.2 on CharXiv-RQ, 44.04 on ChartQAPro, and 52.09 / 49.93 on the single-chart and multi-chart splits of ChartRQA (Chen et al., 21 Jul 2025). These results support the broader claim that chart reasoning benefits from explicit intermediate structure, but they also show that structure can appear as prompting, operator execution, code synthesis, or RL-supervised decomposition.

Beyond charts, diagrammatic intermediates also improve long-horizon planning. “Visualizing Thought” reports that GPT-4o improves from 35.5% to 90.2% in Blocksworld when reasoning through self-generated conceptual diagrams inside a graph-of-thought inference framework (Borazjanizadeh et al., 14 Mar 2025). This does not establish a chart QA result, but it strengthens the wider argument that visualized intermediate structure can materially change reasoning behavior.

6. Limitations, caveats, and open problems

The literature is explicit that structured thought does not eliminate chart reasoning failures. GoT-CQA notes that visual extraction can still fail, complex question parsing is still imperfect, the current Loc/Num/Log set is not fully sufficient, and hard numerical estimation remains challenging (Zhang et al., 2024). These limitations matter because a structured execution graph is only as reliable as its chart encoding and question decomposition.

Programmatic reasoning introduces its own failure mode: numerical hallucination from incorrect symbolic extraction. The Visual Programmability paper gives a failure case in which the model generated wrong data, rendered a wrong chart from that data, and then reasoned consistently from the wrong internal representation (Tang et al., 11 Sep 2025). Its central conclusion is therefore not that code is always superior, but that models must learn when to use code and when direct visual reasoning is safer.

More structure is also not uniformly beneficial. “Table as Thought” improves calendar scheduling, with GPT-4o reaching 74.8 versus 69.4 for Text as Thought, 64.5 for CoT, and 64.0 for Direct, but it does not consistently outperform simpler baselines on GSM8K and MATH500 (Sun et al., 4 Jan 2025). The paper is unusually clear that schema design is hard, that open-source models struggled to complete the structured schema pipeline reliably, and that more structure can introduce overhead and failure modes.

Claims of formal guarantee also require care. “On the Diagram of Thought” argues that topos-theoretic internal logic and DAG structure support consistency and soundness, but the paper itself is best read as offering formal grounding rather than a fully formal completeness/soundness theorem for all model outputs (Zhang et al., 2024). A related methodological boundary appears in chart authoring: “Talk Me Through It” shows that spoken imagined-chart instructions are much longer and structurally richer than typed existing-chart instructions, and that a system trained on spoken imagined-chart data outperforms one trained on typed existing-chart data on voice input and, in the paper’s synthesis, on both voice and text input (Ponochevnyi et al., 21 Jan 2026). This suggests that the distribution of natural chart-thinking traces remains an open data problem.

A plausible synthesis is that Charts-of-Thought is less a single algorithm than a research orientation. Its recurring commitments are explicit intermediate structure, constrained decomposition, and some form of verification or self-checking. Its unresolved questions are equally recurrent: how to choose the right representation for a given chart-question pair, how to ground intermediate states in faithful chart data, how to scale structured supervision without synthetic brittleness, and how to preserve the benefits of explicit thought without incurring excessive complexity.