VisJudge-Bench: Visualization Quality Benchmark

• VisJudge-Bench is a comprehensive benchmark that assesses multimodal LLMs' ability to evaluate data visualizations based on fidelity, expressiveness, and aesthetics.
• The benchmark employs a structured framework with six expert-validated sub-dimensions to measure truthful data representation, effective communication, and visual design.
• Empirical results show that specialized judge models like VisJudge outperform general MLLMs, achieving lower MAE and higher correlation with expert scores.

VisJudge-Bench is a benchmark for evaluating whether multimodal large language models can assess the aesthetics and overall quality of data visualizations in a manner aligned with expert human judgment. It is introduced as the first comprehensive benchmark for evaluating MLLMs’ performance in assessing visualization aesthetics and quality, and it treats visualization assessment as a distinct problem from both natural-image aesthetic assessment and chart-understanding QA because visualization quality depends simultaneously on truthful data representation, communicative effectiveness, and visual design [2510.22373].

1. Conceptual framework and task definition

The benchmark is organized around the three-part framework of Fidelity, Expressiveness, and Aesthetics. In this formulation, a visualization is not judged only as an image, but as an information artifact whose value depends on whether data are faithfully represented, whether patterns are clearly communicated, and whether the design is aesthetically well presented. The paper explicitly positions this framing against natural-image aesthetics benchmarks such as AVA and ArtiMuse, chart-understanding datasets such as ChartQA, PlotQA, and ChartInsights, and visualization evaluation benchmarks such as VisEval and VIS-Shepherd, arguing that none of those directly measures visualization quality as such [2510.22373].

At the task level, the benchmark asks a model to inspect a visualization image, score it on six sub-dimensions using a 1–5 scale, and provide reasoning. The six sub-dimensions are expert-validated and are also grouped into the three higher-level dimensions. The benchmark prompt template requires a JSON object containing scores and rationales for all six dimensions together with an average_score, defined as the mean of the six dimension scores rounded to two decimals.

Higher-level dimension	Sub-dimensions	Role
Fidelity	Data Fidelity	Truthful data representation
Expressiveness	Semantic Readability; Insight Discovery	Communication of information and support for insight
Aesthetics	Design Style; Visual Composition; Color Harmony	Visual quality and design coherence

This design makes VisJudge-Bench neither a generic beauty benchmark nor a chart-reading benchmark. It evaluates whether a model can act as a visualization quality judge. A chart can therefore receive a low score not only because it is visually unattractive, but also because it is misleading, cluttered, or analytically unhelpful. This suggests a broader notion of evaluation in which perceptual, semantic, and design judgments are coupled rather than separated into independent benchmark families.

2. Corpus construction and dataset composition

VisJudge-Bench is built from real-world visualizations gathered from the web. The collection pipeline began with over 300,000 images obtained through Bing Image Search and more than 2,000 automatically generated search keywords derived from over 200 professional visualization terms, more than 30 chart types, quality modifiers such as “professional,” “clean,” “poor,” and “cluttered,” and terminology from over 20 domains including business, finance, healthcare, and education. Automated filtering and perceptual-hash deduplication reduced the pool to 80,210 candidate images. After size filtering, heuristic content screening, and a strict GPT-4o semantic filter restricted to “clean, front-facing, person-free visualization screenshots”, human verification yielded 13,220 valid visualization samples, from which the final 3,090 were selected by stratified random sampling [2510.22373].

The benchmark is balanced across three scenario types: 1,041 single visualizations, 1,024 multiple visualizations, and 1,025 dashboards. It covers 32 subtypes, comprising 22 single-chart subtypes, 5 multi-view subtypes, and 5 dashboard subtypes. Among single visualizations, the subtype distribution includes bar chart (176), pie chart (129), line chart (100), area chart (75), treemap (62), Sankey diagram (61), heatmap (55), scatter plot (49), histogram (48), donut chart (47), funnel chart (45), bubble chart (29), choropleth map (25), radar chart (24), network graph (23), candlestick chart (20), gauge chart (20), box plot (17), point map (12), word cloud (1), violin plot (1), and other single view (22). Multiple visualizations are divided into comparison views (670), small multiples (195), coordinated views (97), overview detail (3), and other multi view (59). Dashboards are divided into analytical dashboards (743), operational dashboards (122), interactive dashboards (91), strategic dashboards (62), and other dashboards (7) [2510.22373].

The score distribution is intentionally broad. Across the full dataset, scores approximately follow a normal distribution with mean 3.13, standard deviation 0.72, and range 1.00–4.89. Single visualizations and multiple visualizations have means of 2.910 and 2.917, while dashboards have mean 3.555. At the sub-dimension level, the means are 3.138 for Data Fidelity, 2.983 for Semantic Readability, 2.983 for Insight Discovery, 2.972 for Design Style, 3.314 for Visual Composition, and 3.369 for Color Harmony. The paper interprets the higher dashboard mean as reflecting the fact that dashboards found in the wild are often more production-ready and polished than isolated charts.

3. Annotation protocol and score governance

Each visualization was initially scored by three independent annotators on all six dimensions. Annotators were recruited via CloudResearch under strict criteria: at least a Bachelor’s degree, preference for Master’s, professional, or doctoral degree holders, historical approval rate 97–100%, 100–10,000 approved projects, native English speakers, activity in the last 180 days, age 20–50, and occupational backgrounds including arts, business, education, finance, STEM, public administration, product design, and technical roles such as developers, designers, analysts, and content creators. Compensation was USD \$10 per hour [2510.22373].

The annotation workflow was highly structured. Tasks were grouped into batches of 15 images, each batch containing 5 single visualizations, 5 multiple visualizations, and 5 dashboards. Because each image had six questions, a task comprised 90 multiple-choice questions and typically required 30–60 minutes. Annotators first received detailed explanations of the three-dimensional framework, then rated each image on the six sub-dimensions using a 1–5 Likert-style scale with explicit descriptive anchors. The question set was not fixed across all images. Instead, GPT-4o first extracted metadata such as chart type and visual elements, after which adaptive question templates instantiated content-specific evaluation questions and 1–5 scoring rubrics.

Quality control proceeded in two layers. During crowdsourcing, the interface embedded validation checks using chart-pair questions where the better or worse chart was visually obvious. After initial annotation, disagreement analysis identified high-disagreement samples where the standard deviation across the three annotators exceeded 1.0. Candidate resolution strategies included outlier removal, malicious scoring filtering, and sub-dimension bias correction. Outlier removal applied when two annotators’ scores were close with absolute difference (\leq 2.0) and the third was far from both with absolute difference (> 1.5) from each. Malicious scoring detection flagged cases where an annotator assigned identical scores across all six dimensions. Sub-dimension bias correction flagged any score deviating by more than 2.0 from the average of the other two annotators. These suggestions did not determine labels automatically: a team of three visualization experts independently reviewed disputed samples and the candidate strategies, then selected, modified, or rejected them, with consensus discussion on difficult cases [2510.22373].

All final scores are therefore expert-validated. The appendix reports that all 3,090 samples underwent expert review, 2,606 samples (84.3%) included alternative results for cross-validation, and 1,792 samples (58.0%) had their scores refined through expert calibration. The paper does not report a conventional inter-annotator agreement statistic such as Cohen’s (\kappa), Krippendorff’s (\alpha), or Fleiss’ (\kappa); instead it treats the multilayer adjudication process and expert consensus as the mechanism for label reliability. This suggests a benchmark-governance model centered on expert calibration rather than purely statistical agreement reporting.

4. Evaluation protocol, metrics, and empirical findings

For training and evaluation, the dataset is split by stratified sampling into 70% training, 10% validation, and 20% test, corresponding to 2,163 / 279 / 648 samples. The test set contains 231 single visualizations, 209 multiple visualizations, and 208 dashboards. It preserves the overall score distribution, with mean 3.13, standard deviation 0.72, and range 1.11–4.89. Quality bands in the test set are 7.4% for scores 1.0–2.0, 31.5% for 2.0–3.0, 49.5% for 3.0–4.0, and 11.6% for 4.0–5.0 [2510.22373].

Each evaluated model is prompted under the same framework and asked to output 1–5 scores plus justifications. To reduce stochasticity, each model is run three times and the results are averaged. The reported metrics are Mean Absolute Error (MAE), Mean Squared Error (MSE), and Pearson correlation coefficient. In the notation given in the benchmark description,

[
\mathrm{MAE} = \frac{1}{N} \sum_{i=1}^{N} |\hat{y}_i - y_i|
]

[
\mathrm{MSE} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2
]

[
r = \frac{\sum_{i=1}^{N} (\hat{y}i - \bar{\hat{y}})(y_i - \bar{y})}
{\sqrt{\sum{i=1}^{N} (\hat{y}i - \bar{\hat{y}})^2}\sqrt{\sum{i=1}^{N} (y_i - \bar{y})^2}}
]

where (\hat{y}_i) is the model prediction and (y_i) is the human score. Lower MAE and MSE are better; higher correlation is better.

The benchmark evaluates seven general-purpose MLLMs plus the specialized VisJudge model. The headline finding is that even strong general MLLMs remain substantially below expert judgment. GPT-5, the strongest general MLLM in the abstract, achieves MAE 0.551 and correlation 0.429, whereas VisJudge achieves MAE 0.442 and correlation 0.681 [2510.22373].

Model	MAE	Corr.
Claude-3.5-Sonnet	0.823	0.395
Claude-4-Sonnet	0.618	0.470
Gemini-2.0-Flash	0.680	0.395
Gemini-2.5-Pro	0.661	0.266
GPT-4o	0.609	0.482
GPT-5	0.551	0.429
Qwen2.5-VL-7B	1.048	0.322
VisJudge	0.442	0.681

Several patterns are important. First, GPT-5 has the best MAE among general MLLMs, but GPT-4o has the highest overall correlation among commercial baselines. Second, VisJudge improves over both: compared with GPT-5, MAE drops from 0.551 to 0.442, a 19.8% reduction, and compared with GPT-4o, correlation rises from 0.482 to 0.681, a 41.3% improvement. Third, performance is uneven across sub-dimensions. For GPT-5, MAEs are 0.861 for Fidelity, 0.780 for Semantic Readability, 0.776 for Insight Discovery, 0.648 for Design Style, 0.698 for Composition, and 0.682 for Color; correlations are 0.256, 0.438, 0.383, 0.463, 0.277, and 0.295 respectively. The paper states that all models struggle significantly on Aesthetics dimensions, with average MAE around 0.76 and most correlations below 0.3, except Design Style at 0.44.

The benchmark also exposes systematic score bias. Human experts have average score (\mu = 3.13). Several models show inflation: Qwen2.5-VL-7B at 3.89, Claude-3.5-Sonnet at 3.87, Gemini-2.0-Flash at 3.64, GPT-4o at 3.53, Claude-4-Sonnet at 3.56, and GPT-5 at 3.36. Gemini-2.5-Pro is described as overly conservative with (\mu = 3.02). VisJudge nearly matches the human score distribution with (\mu = 3.11). Complexity analysis adds a further result: performance degrades from single visualizations to multiple visualizations to dashboards, and in dashboards some baseline models even show negative Data Fidelity correlations, including Claude-3.5-Sonnet at (-0.031) and GPT-5 at (-0.013).

5. VisJudge as a specialized judge model

The benchmark is not only evaluative. It is also used to train the specialized VisJudge model, which is built by fine-tuning Qwen2.5-VL-7B-Instruct on the benchmark’s human-annotated data. The model takes a visualization image together with the structured evaluation prompt based on the Fidelity, Expressiveness, and Aesthetics rubric, and it outputs a structured JSON containing per-dimension scores, rationales, and the average score [2510.22373].

Training uses reinforcement learning with the GRPO algorithm together with LoRA adaptation. The reported configuration is: 5 epochs on four NVIDIA A6000 GPUs (48 GB each), learning rate (10^{-5}), AdamW optimizer, weight decay 0.01, cosine annealing scheduler with warmup ratio 0.1, LoRA rank and alpha 128 applied to all linear layers, GRPO beta 0.01, global batch size 16, and bfloat16 mixed precision. The software stack uses SWIFT with PyTorch and DeepSpeed ZeRO Stage 2.

The reward function is defined explicitly as a composite of accuracy and format rewards:

[
R_{\text{composite}} = 0.9 R_{\text{acc}} + 0.1 R_{\text{format}}
]

For a single score prediction, the accuracy reward is

[
R_{\text{acc_single}} = \exp\left(-\frac{\left|\text{score}{\text{predicted}} - \text{score}{\text{ground-truth}}\right|}{0.5}\right)
]

and the final accuracy reward is the average over the six dimension scores and the overall average score. The format reward is binary: it is 1.0 if the output contains all required fields in the expected JSON structure and 0 otherwise. This training design makes structure compliance an explicit optimization target rather than only a parsing convenience.

The paper also studies VisJudge-Bench as a scaling resource. With a single training epoch and progressively larger subsets, human-model correlation rises from 0.30 to 0.65 at 2,442 samples, approximately following a logarithmic trend with (R^2 = 0.628). Prediction error decreases smoothly: MAE drops from 1.05 to 0.45 with (R^2 = 0.984), and MSE from 1.55 to 0.30 with (R^2 = 0.992). The 500–1,000 sample regime yields 45% of total correlation improvement, after which improvements continue with diminishing returns. This suggests that the benchmark functions not only as an evaluation set but also as a domain-specific training corpus for visualization judges.

6. Significance, limitations, and relation to judge-benchmark research

VisJudge-Bench fills a missing layer in the visualization ecosystem by benchmarking MLLMs as visualization critics rather than as visualization generators or chart readers. The paper identifies practical use cases including automatic visualization recommendation, chart critique, dashboard evaluation, quality control for generated visualizations, and potentially training judge models to support NL2VIS systems. Its six-dimensional labels provide both a holistic score and a structured error surface, which is useful for downstream analysis [2510.22373].

The benchmark also has clear limitations. First, judgments of visualization aesthetics and quality inevitably reflect human preferences and may encode bias. Second, the data come from web-crawled screenshots and therefore reflect the domains, tools, and publication practices visible on the web rather than a controlled sample of all visualization practices. Third, dashboards in the dataset tend to have higher quality than single or multi-view charts, which may influence model calibration. Fourth, the benchmark is built around scalar rating prediction rather than pairwise preference learning or free-form critique alone, although rationales are generated; the paper does not present a dedicated pairwise-comparison benchmark. These limitations matter because visualization judgment has both objective and subjective components, and the current benchmark operationalizes that mixture through expert consensus rather than through a formal theory of visual quality.

Within the broader judge-evaluation literature, VisJudge-Bench occupies a specific niche. Related work has emphasized multilingual judge reliability under position bias, verbosity bias, order inconsistency, and cross-lingual degradation [2606.22329]; adaptive, debiased, and consistent judges that model judging as a conditional policy and enforce pointwise versus pairwise consistency [2602.06625]; and psychometric characterization of judges through dark current, stable cross-sensitivity, positional false preference, target sensitivity, and criterion [2606.15610]. Other benchmarks stress contextual evidence grounding in RAG and summarization [2503.15620], video-grounded evaluation where direct access to the video matters [2509.21451], capability-oriented multimodal judge benchmarking through pairwise CoT comparison, length bias avoidance, and process error detection [2603.00546], and constraint-level judge evaluation with yes/partial/no labels plus inconsistency metrics [2605.03858]. This suggests that VisJudge-Bench is best understood as a domain-specific benchmark for visualization-quality score prediction, whereas much of the adjacent literature focuses more directly on judge bias, invariance, calibration, consistency, or perturbation robustness.

In that broader landscape, VisJudge-Bench establishes a benchmark and training resource for expert-aligned visualization judgment, but not yet a full reliability audit framework. Its distinctive contribution is the claim that visualization judgment requires simultaneous assessment of truthfulness, communicative value, and visual design, and that current MLLMs remain substantially below expert standards on that combined task.