ChartAgent: Multimodal Chart QA
- The paper introduces ChartAgent, a multimodal agent that decomposes chart questions into visual subtasks for accurate, spatial reasoning.
- It employs a ReAct-style loop with specialized tools (e.g., cropping, annotation, interpolation) to process both annotated and unannotated charts.
- The approach achieves state-of-the-art results on ChartBench and ChartX, with up to 17.31% absolute gain on unannotated and numeric queries.
ChartAgent is a multimodal agent for chart question answering that explicitly performs visual reasoning directly within a chart’s spatial domain. Instead of relying on textual chain-of-thought alone, it iteratively decomposes a chart question into visual subtasks and executes chart-specific actions such as drawing annotations, cropping regions, localizing axes, segmenting bars or pie slices, extracting colors, and interpolating values from pixels. The framework was introduced to address the sharp decline of multimodal LLMs on unannotated charts and numerically intensive queries, and it reports state-of-the-art accuracy on ChartBench and ChartX, including up to 16.07% absolute gain overall and 17.31% on unannotated, numerically intensive queries (Kaur et al., 6 Oct 2025).
1. Problem setting and conceptual basis
ChartAgent is designed for complex chart question answering in settings where answers cannot be recovered reliably through OCR shortcuts. The central distinction is between annotated charts, which contain explicit textual values near marks, and unannotated charts, which do not. On annotated charts, many multimodal systems can rely on OCR plus lightweight language reasoning. On unannotated charts, however, the answer must be inferred from bar heights, line positions, slice areas, radial lengths, boxplot quartiles, axis scales, and legends. ChartAgent is explicitly motivated by this gap and frames it as a failure of visual grounding rather than a failure of generic language reasoning (Kaur et al., 6 Oct 2025).
A defining feature of the framework is its rejection of purely textual chain-of-thought as the main substrate for chart reasoning. Instead, it reasons by acting on the image: cropping a legend, annotating legend entries, isolating a bar, segmenting a ring chart, localizing an axis, or converting pixel coordinates into numeric values. This produces explicit intermediate visual evidence and creates a closed-loop perception-and-correction process. The agent therefore mirrors a human chart-reading strategy more closely than a one-shot multimodal answer generator.
This emphasis on visual grounding also explains why the framework is strongest on unannotated and numeric chart QA. The paper repeatedly characterizes ChartAgent as one of the first systems to demonstrate visually grounded reasoning for chart understanding using tool-augmented multimodal agents, with the chart image treated as an interactive spatial object rather than a passive prompt (Kaur et al., 6 Oct 2025).
2. Architecture and reasoning loop
The system begins with an LLM-based orchestrator, typically GPT-4o (gpt-4o-2024-08-06), that extracts chart metadata such as chart type, title, legend details, axis labels, tick marks, whether the chart is annotated or unannotated, and a concise visual description. This metadata supports both routing and conditioning. If the chart is annotated or the query is mostly qualitative, ChartAgent can route directly to the base multimodal model. If the chart is unannotated and numerically demanding, it activates the full agentic loop (Kaur et al., 6 Oct 2025).
The core inference procedure is a ReAct-style multimodal loop. At each step, the model evaluates the current state, derives the next sub-goal, selects one chart-tool action, receives the resulting observation, and updates the state. Sub-goals are primarily visual perception tasks, though they may culminate in arithmetic operations or comparisons. The loop repeats until the model outputs a final answer or reaches the maximum of 15 iterations. If satisfactory tool-based reasoning cannot be achieved after multiple tries, the system falls back to direct MLLM reasoning.
Chart-type-specific few-shot in-context examples further shape behavior. The framework retrieves 1–2 complete ReAct trajectories conditioned on chart type, when such examples exist, so that the model is shown how a successful reasoning process typically unfolds for that chart family. This is a prompt-level policy prior rather than a learned action policy.
Implementation is deliberately modular. The agent stack is built with AutoGen 0.2.26 in a Python 3.9 environment, uses temperature 0.0, and was run on AWS g4dn.xlarge with a single NVIDIA T4 GPU with 16 GB. The base multimodal model acts as planner, controller, and final verbalizer; perception is delegated to specialized external tools. The paper characterizes the entire framework as training-free at the agent level, with no additional supervised training of the policy itself (Kaur et al., 6 Oct 2025).
3. Tool library and spatial operations
ChartAgent’s main systems contribution is a chart-specific tool library divided into universal chart tools and chart-specific tools. These tools expose an action space over chart primitives rather than generic image crops or zooms.
| Category | Tools | Function |
|---|---|---|
| Universal tools | annotate_legend, get_marker_rgb, clean_chart_image, segment_and_mark, axis_localizer, interpolate_pixel_to_value, arithmetic |
Legend parsing, color grounding, cleanup, segmentation, axis localization, value interpolation, numeric operations |
| Pie / treemap | compute_segment_area |
Segment-area estimation by counting elements or pixels |
| Bar / combination | get_bar, compute_bar_height |
Bar localization and pixel-to-value conversion |
| Box plot | get_boxplot, compute_boxplot_entity |
Boxplot detection and extraction of max, min, median, Q1, Q3, range, and interquartile range |
| Line / area / scatter | get_edgepoints |
Edge-point localization for line segments, dots, and related structures |
| Radial charts | get_radial, analyze_radial_geometry, estimate_radial_value |
Radial segment detection and value estimation |
The perception backends are also explicit. Segment Anything v1 (ViT-H) is used for segmentation, with 641M parameters and 2.56 GB footprint; Tesseract OCR and EasyOCR are used for text, ticks, and legends. These are external modules rather than jointly trained components. Their outputs are visualized and returned to the agent as interpretable observations, which is what enables visual self-verification (Kaur et al., 6 Oct 2025).
This self-verification mechanism is central. If a segmentation mask is poor, a legend color does not match the intended series, a bar height becomes negative, or interpolation fails, the agent can retry with modified parameters, switch tools, or eventually fall back. In a manual analysis of 30 ChartBench trajectories, 50% of cases required no recovery; in the remaining 50%, the agent detected unsatisfactory tool outputs, and among those recovery-needed cases it recovered successfully 70% of the time. Unresolved tool failures contributed 15% overall error in that sample (Kaur et al., 6 Oct 2025).
4. Benchmarks, performance, and empirical profile
The framework is evaluated on ChartBench and ChartX, two benchmarks chosen because they emphasize unannotated, visually grounded, numerically intensive chart QA. ChartBench contains 3,800 chart-QA pairs, with 76.2% unannotated and 96.7% numeric QA. ChartX contains 1,152 chart-QA pairs, with 61.7% unannotated and 71.9% numeric QA (Kaur et al., 6 Oct 2025).
| Benchmark | Data profile | Main result |
|---|---|---|
| ChartBench | 3,800 pairs; 76.2% unannotated; 96.7% numeric | 71.39% overall; 60.81% unannotated; 70.91% numeric |
| ChartX | 1,152 pairs; 61.7% unannotated; 71.9% numeric | 59.69% overall; 44.16% unannotated; 55.93% numeric |
On ChartBench, ChartAgent achieves 71.39% overall, 94.33% on annotated charts, 60.81% on unannotated charts, 70.91% on numeric QA, and 91.00% on relationship QA. On ChartX, it achieves 59.69% overall, 84.84% on annotated charts, 44.16% on unannotated charts, 55.93% on numeric QA, and 69.14% on VC/GC QA. The headline gains are on ChartBench: +16.07% absolute overall, +17.31% absolute gain on unannotated, and +15.02% absolute gain on numeric QA relative to the best prior baselines reported in the main table. On ChartX, the overall gain is +2.83% (Kaur et al., 6 Oct 2025).
Type-wise, the strongest gains on unannotated ChartBench charts appear on bar charts, box plots, combination charts, and pie charts. The paper summarizes gains up to +65% on Bar, +69% on Box, up to +23% on Combination, and up to +62% on Pie. Weaknesses remain on 3D charts and radar charts. On ChartX, the framework is notably strong on Bubble, Ring, and Treemap, but still struggles on Multi-Axes and Radar.
Complexity analysis shows the same pattern. On unannotated ChartBench, ChartAgent reaches 83.98% on visually Easy, 56.77% on visually Medium, and 17.92% on visually Hard; on reasoning difficulty it reaches 71.33% on Easy, 41.68% on Medium, and 28.52% on Hard. The visually Hard regime is the only one where it is not best. This concentration of failures is consistent with the paper’s later failure analysis.
Ablation results underscore that the gains are specifically agentic and specifically chart-centric. With the same GPT-4o backbone, ReAct (No Tools) scores 38.84% overall and 19.46% on unannotated+numeric QA; ReAct + Natural Image Tools scores 41.35% overall and 20.50% on unannotated+numeric QA; full ChartAgent scores 71.39% overall and 58.29% on unannotated+numeric QA. The plug-and-play analysis also reports absolute gains on unannotated+numeric QA of +26.7% with GPT-4o, +23.9% with GPT-4o-mini, +28.4% with Claude 3 Haiku, and +12.2% with Pixtral, supporting the claim that ChartAgent is a framework rather than a single-model phenomenon (Kaur et al., 6 Oct 2025).
5. Failure modes, scope conditions, and operational constraints
The paper’s failure analysis is dominated by perception-driven errors. These include OCR blocked by overlays or dense elements, poor color contrast, legends occluding key chart regions, invisible lines or markers blending into the background, segmentation failures from axis overlap, overlapping series, and distorted or inconsistent axes in 3D or multi-axis charts. A second class involves reasoning failures, such as choosing the wrong measurement strategy, ambiguity in multi-ring pie charts, duplicated labels across hierarchy levels, subtype confusion such as stacked versus overlay area charts, and unit mismatches (Kaur et al., 6 Oct 2025).
Fallback to direct MLLM reasoning is reported as rare (<10%) in a manual sample. It is typically triggered when bar heights become negative or axis-inconsistent, OCR returns None, or interpolation and edge-point tools return empty or inconsistent values. This indicates that most errors arise before or during grounded measurement rather than after answer verbalization.
The framework also has explicit scope boundaries. It is evaluated only for QA, not for summarization, fact-checking, or broader chart-document interaction. It operates in a single-chart setting rather than across multi-chart documents or dashboards. The paper does not use multimodal in-context learning, only textual ICL. It also reports a substantial latency gap: GPT-4o + CoT takes about 12s/query, whereas a full ChartAgent trajectory takes about 125s/query (Kaur et al., 6 Oct 2025).
These limitations are especially salient when chart interaction moves beyond static screenshots. ChartAct, an interactive benchmark for dynamic chart understanding, shows that real chart environments often require hovering, clicking, zooming, or dragging to expose critical evidence. It collects 673 dynamic charts from 8 real chart websites, with 1,440 QA samples instantiated in both Dynamic Chart and Dashboard Chart settings, and reports that even the strongest model, Claude-Opus-4.7, reaches an average success rate of 84.5%, while most models remain below 60% (Huang et al., 26 May 2026). This establishes a neighboring frontier that ChartAgent itself does not yet address.
6. Position within chart-understanding research
ChartAgent belongs to a broader shift from chart recognition toward structured, grounded, and executable chart understanding. Earlier systems concentrated on extracting machine-readable structure from static charts. CHARTER framed the problem as end-to-end conversion of document charts into machine-readable tabular data, combining page-level detection, chart-element localization, OCR, and chart-type-specific reconstruction logic (Shtok et al., 2021). ChartAssistant later argued for chart-to-table parsing as the central bridge between chart images and language, then extended this bridge with multitask instruction tuning for QA, referring interaction, and summarization (Meng et al., 2024). OneChart pushed structural extraction further with a numerically optimized auxiliary token and a self-evaluation mechanism for chart parsing reliability (Chen et al., 2024). ChartReader likewise sought a unified framework for chart derendering and comprehension without heuristic rules (Cheng et al., 2023).
A parallel strand emphasizes executable representations. ChartGen formulates chart understanding as chart-to-code reconstruction, producing 222.5K unique chart image-code pairs and a held-out 4.3K chart redrawing benchmark (Kondic et al., 31 May 2025). CycleChart makes a related schema-centric argument by aligning chart generation, schema parsing, data parsing, and QA around a single consistent lifecycle (Deng et al., 22 Dec 2025). These works share with ChartAgent a preference for grounded intermediate structure rather than purely textual descriptions, but they typically do not operate as inference-time visual agents.
There is also a complementary line on chart editing and design control. ChartReformer edits chart images from natural-language prompts by reconstructing an underlying data table and visual attributes, then re-rendering through a plotting engine (Yan et al., 2024). ChartOptimiser treats chart design as a black-box optimization problem over an eight-dimensional design space for bar charts, optimized for tasks such as finding the extreme value, retrieving a value, comparing two values, and computing a derived value (Wang et al., 14 Apr 2025). These systems address manipulation and optimization rather than question answering, but they reinforce the importance of structured chart representations and task-conditioned operations.
Taken together, this literature places ChartAgent as a specifically agentic instantiation of chart understanding: earlier work emphasizes chart-to-table, chart-to-dict, chart-to-schema, or chart-to-code; ChartAgent instead makes the spatial chart itself the substrate for iterative action. This suggests a broader taxonomy in which chart intelligence spans structured extraction, instruction tuning, executable reconstruction, editing, optimization, and interactive tool-augmented reasoning.