- The paper introduces a bilingual benchmark covering eight chart families and three visual scenarios that standardizes diverse output formats.
- It employs a deterministic normalization protocol to convert heterogeneous representations into unified semantic structures for numeric and diagrammatic charts.
- Evaluation of 26 models reveals robust performance by large MLLMs while exposing limitations in parsing diagrammatic and hand-drawn charts under noisy conditions.
Motivation and Limitations of Existing Chart Parsing Benchmarks
While chart parsing has seen significant advances with the expansion of multimodal LLMs (MLLMs), systematic and generalizable evaluation protocols have remained restricted. Prior benchmarks are limited along several axes: (i) constrained chart family diversity (focusing mainly on bar, line, and pie charts); (ii) lack of diagrammatic structure coverage (neglecting flowcharts and mind maps); and (iii) absence of challenging real-world scenarios such as printed or hand-drawn imagery. Additionally, heterogeneous output formats—including Markdown, JSON, CSV, SVG, and code—have created an evaluation bottleneck, rendering cross-model comparisons intractable. Without normalization across representations and comprehensive chart-type coverage, model improvements cannot be reliably measured.
ChartArena: Benchmark Design and Data Collection
ChartArena explicitly addresses these deficiencies by establishing a large-scale, bilingual (Chinese and English) benchmark comprising eight chart families: bar, line, pie, radar, box plot, combination chart, flowchart, and mind map. Each family is represented under three visual scenarios (digital rendering, printed photo, hand-drawn photo) with structurally consistent annotation, yielding 2,400 chart instances. The dataset’s construction pipeline is human–agent collaborative: draft annotations are generated by MLLMs and iteratively refined via thorough human multi-stage verification for consistency and fidelity. Hard-to-annotate scenarios, especially in printed and hand-drawn categories, are deliberately oversampled to induce robust evaluation metrics in non-ideal visual conditions. This approach is orthogonal to previous benchmarks (e.g., PlotQA-SE, MMC-Bench, ChartX-SE) that neglect low-resource chart types and real-world acquisition noise.
A critical innovation is the deterministic normalization protocol enabling fair evaluation independent of chart type or output surface format. ChartArena canonicalizes heterogeneous outputs into two semantic representations: a normalized triple view for numeric charts and a directed graph view for diagrammatic charts. All models, regardless of their preferred serialization, are routed through adapters that extract and map core data (e.g., values, headers, connections) to these canonical structures. For numeric charts, outputs are reduced to (header, entity, value) tuples, with robust lexical and numeric canonicalization. For diagrammatic charts, directed labeled graphs are constructed from all graphical output languages (e.g., Mermaid, DOT, PlantUML).
Scoring is based on structure-aware metrics:
- Intersection-over-Union (IoU) over triples for numeric charts, accounting for minor OCR and rounding imprecisions via tolerance-based Levenshtein and relative numerical thresholds.
- Hungarian-matched node and edge alignment for graphs, with edge topology emphasized, and a variant for tree structures (mind maps) rewarding partial structural correctness.
Metrics include strict Exact Match (EM) and mean Average Precision (mAP) under several tolerance regimes, systematically capturing both all-or-nothing semantic recovery and granularity of partial correctness.
Large-Scale Model Evaluation and Analysis
Twenty-six leading models are evaluated: 16 general-purpose MLLMs (including proprietary and state-of-the-art open-source models such as Gemini 3.1 Pro, Qwen3.5-35B-A3B, Kimi K2.5), three document parsing MLLMs, and seven dedicated chart parsers. Notable results:
- Gemini 3.1 Pro achieves the highest overall mAP, with strong performance in both English and Chinese and robust generalization across visual scenarios.
- The top open-source MLLMs (Qwen3.5-35B-A3B, Kimi K2.5) approach proprietary models in benchmark performance, indicating rapid community progress.
- Radar charts and all hand-drawn inputs remain universally challenging: even best-in-class models show substantial performance degradation relative to clean digital renderings, highlighting current model limitations in extracting orientation- and layout-sensitive information under noise.
- Diagrammatic charts, particularly mind maps and flowcharts, are a bottleneck for both document parsing MLLMs and expert chart parsers, underscoring the need for models with enriched topological and logical reasoning capability.
- Expert chart parsers consistently show narrow coverage and inability to generalize, with most unable to process diagrammatic structures or scenarios outside their very limited training slices.
Results demonstrate clear failure modes: top MLLMs often yield conservative outputs (e.g., blank entries for ambiguous cases), whereas smaller models “hallucinate” plausible but incorrect values, confirming distinct types of semantic errors under adverse visual conditions.
Evaluation Protocol Validation and Implications
The protocol’s normalization reveals that most standard output formats yield stable and comparable scores; however, SVG and PlantUML exhibit decreased effectiveness due to fundamental mismatches with the underlying chart’s semantic requirements. This diagnostic feature provides actionable guidance for future model and template design. The benchmark's results reveal that existing models lack robustness to real-world distribution drift, and that diagram-level reasoning is not yet reliably achieved by current architectures.
Theoretical and Practical Implications
Practically, ChartArena establishes a unified, rigorous basis for benchmarking and accelerating cross-model progress in chart parsing, useful across scientific, business, and accessibility applications. Theoretically, its findings expose open research problems: generalizing structured parsing and semantic normalization across noisy, visually diverse, and topologically complex chart families remains unsolved. The clear superiority of large-parameter MLLMs on diagrammatic charts points to implicit reliance on broad world-knowledge priors—suggesting that data scale and pretraining diversity are minimum requirements for robust topological understanding.
Future developments will likely require:
- Expansion to multi-page, multi-modal, and scatter plot scenarios,
- Model architectures with explicit reasoning over hierarchical and graph structure,
- Training procedures incorporating structured feedback and adversarial, real-world augmented data.
Conclusion
ChartArena fills critical gaps in the chart parsing evaluation landscape by unifying diverse chart types, visual sources, and output formats under a robust, format-agnostic protocol. Its thorough modeling, detailed annotation, and nuanced metrics uncover capability gaps in all current state-of-the-art systems, especially under visually and structurally challenging conditions. As both a resource and methodology, ChartArena is poised to become the standard benchmark for both algorithmic advances and cross-model comparison in structured visual understanding tasks.
Reference:
"ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats" (2606.01348)