---
title: 'VisJudge-Bench: Visualization Quality Benchmark'
url: https://www.emergentmind.com/topics/visjudge-bench
type: topic
---

# VisJudge-Bench: Visualization Quality Benchmark

VisJudge-Bench is a benchmark for evaluating whether multimodal large language models can assess the aesthetics and overall quality of data visualizations in a manner aligned with expert human judgment. It is introduced as the first comprehensive benchmark for evaluating MLLMs’ performance in assessing visualization aesthetics and quality, and it treats visualization assessment as a distinct problem from both natural-image aesthetic assessment and chart-understanding QA because visualization quality depends simultaneously on truthful data representation, communicative effectiveness, and visual design [2510.22373].

## 1. Conceptual framework and task definition

The benchmark is organized around the three-part framework of **Fidelity, Expressiveness, and Aesthetics**. In this formulation, a visualization is not judged only as an image, but as an information artifact whose value depends on whether data are faithfully represented, whether patterns are clearly communicated, and whether the design is aesthetically well presented. The paper explicitly positions this framing against natural-image aesthetics benchmarks such as AVA and ArtiMuse, chart-understanding datasets such as ChartQA, PlotQA, and ChartInsights, and visualization evaluation benchmarks such as VisEval and VIS-Shepherd, arguing that none of those directly measures visualization quality as such [2510.22373].

At the task level, the benchmark asks a model to inspect a visualization image, score it on six sub-dimensions using a 1–5 scale, and provide reasoning. The six sub-dimensions are expert-validated and are also grouped into the three higher-level dimensions. The benchmark prompt template requires a JSON object containing scores and rationales for all six dimensions together with an `average_score`, defined as the mean of the six dimension scores rounded to two decimals.

| Higher-level dimension | Sub-dimensions | Role |
|---|---|---|
| Fidelity | Data Fidelity | Truthful data representation |
| Expressiveness | Semantic Readability; Insight Discovery | Communication of information and support for insight |
| Aesthetics | Design Style; Visual Composition; Color Harmony | Visual quality and design coherence |

This design makes VisJudge-Bench neither a generic beauty benchmark nor a chart-reading benchmark. It evaluates whether a model can act as a **visualization quality judge**. A chart can therefore receive a low score not only because it is visually unattractive, but also because it is misleading, cluttered, or analytically unhelpful. This suggests a broader notion of evaluation in which perceptual, semantic, and design judgments are coupled rather than separated into independent benchmark families.

## 2. Corpus construction and dataset composition

VisJudge-Bench is built from real-world visualizations gathered from the web. The collection pipeline began with **over 300,000 images** obtained through Bing Image Search and more than **2,000 automatically generated search keywords** derived from over **200 professional visualization terms**, more than **30 chart types**, quality modifiers such as “professional,” “clean,” “poor,” and “cluttered,” and terminology from over **20 domains** including business, finance, healthcare, and education. Automated filtering and perceptual-hash deduplication reduced the pool to **80,210 candidate images**. After size filtering, heuristic content screening, and a strict GPT-4o semantic filter restricted to **“clean, front-facing, person-free visualization screenshots”**, human verification yielded **13,220 valid visualization samples**, from which the final **3,090** were selected by stratified random sampling [2510.22373].

The benchmark is balanced across three scenario types: **1,041 single visualizations**, **1,024 multiple visualizations**, and **1,025 dashboards**. It covers **32 subtypes**, comprising **22 single-chart subtypes**, **5 multi-view subtypes**, and **5 dashboard subtypes**. Among single visualizations, the subtype distribution includes bar chart (**176**), pie chart (**129**), line chart (**100**), area chart (**75**), treemap (**62**), Sankey diagram (**61**), heatmap (**55**), scatter plot (**49**), histogram (**48**), donut chart (**47**), funnel chart (**45**), bubble chart (**29**), choropleth map (**25**), radar chart (**24**), network graph (**23**), candlestick chart (**20**), gauge chart (**20**), box plot (**17**), point map (**12**), word cloud (**1**), violin plot (**1**), and other single view (**22**). Multiple visualizations are divided into comparison views (**670**), small multiples (**195**), coordinated views (**97**), overview detail (**3**), and other multi view (**59**). Dashboards are divided into analytical dashboards (**743**), operational dashboards (**122**), interactive dashboards (**91**), strategic dashboards (**62**), and other dashboards (**7**) [2510.22373].

The score distribution is intentionally broad. Across the full dataset, scores approximately follow a normal distribution with mean **3.13**, standard deviation **0.72**, and range **1.00–4.89**. Single visualizations and multiple visualizations have means of **2.910** and **2.917**, while dashboards have mean **3.555**. At the sub-dimension level, the means are **3.138** for Data Fidelity, **2.983** for Semantic Readability, **2.983** for Insight Discovery, **2.972** for Design Style, **3.314** for Visual Composition, and **3.369** for Color Harmony. The paper interprets the higher dashboard mean as reflecting the fact that dashboards found in the wild are often more production-ready and polished than isolated charts.

## 3. Annotation protocol and score governance

Each visualization was initially scored by **three independent annotators** on all six dimensions. Annotators were recruited via CloudResearch under strict criteria: at least a Bachelor’s degree, preference for Master’s, professional, or doctoral degree holders, historical approval rate **97–100%**, **100–10,000** approved projects, native English speakers, activity in the last **180 days**, age **20–50**, and occupational backgrounds including arts, business, education, finance, STEM, public administration, product design, and technical roles such as developers, designers, analysts, and content creators. Compensation was **USD \$10 per hour** [2510.22373].

The annotation workflow was highly structured. Tasks were grouped into batches of **15 images**, each batch containing **5** single visualizations, **5** multiple visualizations, and **5** dashboards. Because each image had six questions, a task comprised **90 multiple-choice questions** and typically required **30–60 minutes**. Annotators first received detailed explanations of the three-dimensional framework, then rated each image on the six sub-dimensions using a 1–5 Likert-style scale with explicit descriptive anchors. The question set was not fixed across all images. Instead, GPT-4o first extracted metadata such as chart type and visual elements, after which adaptive question templates instantiated content-specific evaluation questions and 1–5 scoring rubrics.

Quality control proceeded in two layers. During crowdsourcing, the interface embedded validation checks using chart-pair questions where the better or worse chart was visually obvious. After initial annotation, disagreement analysis identified **high-disagreement samples** where the standard deviation across the three annotators exceeded **1.0**. Candidate resolution strategies included outlier removal, malicious scoring filtering, and sub-dimension bias correction. Outlier removal applied when two annotators’ scores were close with absolute difference **\(\leq 2.0\)** and the third was far from both with absolute difference **\(> 1.5\)** from each. Malicious scoring detection flagged cases where an annotator assigned identical scores across all six dimensions. Sub-dimension bias correction flagged any score deviating by more than **2.0** from the average of the other two annotators. These suggestions did not determine labels automatically: a team of **three visualization experts** independently reviewed disputed samples and the candidate strategies, then selected, modified, or rejected them, with consensus discussion on difficult cases [2510.22373].

All final scores are therefore expert-validated. The appendix reports that all **3,090** samples underwent expert review, **2,606** samples (**84.3%**) included alternative results for cross-validation, and **1,792** samples (**58.0%**) had their scores refined through expert calibration. The paper does not report a conventional inter-annotator agreement statistic such as Cohen’s \(\kappa\), Krippendorff’s \(\alpha\), or Fleiss’ \(\kappa\); instead it treats the multilayer adjudication process and expert consensus as the mechanism for label reliability. This suggests a benchmark-governance model centered on expert calibration rather than purely statistical agreement reporting.

## 4. Evaluation protocol, metrics, and empirical findings

For training and evaluation, the dataset is split by stratified sampling into **70% training**, **10% validation**, and **20% test**, corresponding to **2,163 / 279 / 648** samples. The test set contains **231** single visualizations, **209** multiple visualizations, and **208** dashboards. It preserves the overall score distribution, with mean **3.13**, standard deviation **0.72**, and range **1.11–4.89**. Quality bands in the test set are **7.4%** for scores **1.0–2.0**, **31.5%** for **2.0–3.0**, **49.5%** for **3.0–4.0**, and **11.6%** for **4.0–5.0** [2510.22373].

Each evaluated model is prompted under the same framework and asked to output 1–5 scores plus justifications. To reduce stochasticity, each model is run **three times** and the results are averaged. The reported metrics are **Mean Absolute Error (MAE)**, **Mean Squared Error (MSE)**, and **Pearson correlation coefficient**. In the notation given in the benchmark description,

\[
\mathrm{MAE} = \frac{1}{N} \sum_{i=1}^{N} |\hat{y}_i - y_i|
\]

\[
\mathrm{MSE} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2
\]

\[
r = \frac{\sum_{i=1}^{N} (\hat{y}_i - \bar{\hat{y}})(y_i - \bar{y})}
{\sqrt{\sum_{i=1}^{N} (\hat{y}_i - \bar{\hat{y}})^2}\sqrt{\sum_{i=1}^{N} (y_i - \bar{y})^2}}
\]

where \(\hat{y}_i\) is the model prediction and \(y_i\) is the human score. Lower MAE and MSE are better; higher correlation is better.

The benchmark evaluates seven general-purpose MLLMs plus the specialized VisJudge model. The headline finding is that even strong general MLLMs remain substantially below expert judgment. GPT-5, the strongest general MLLM in the abstract, achieves **MAE 0.551** and **correlation 0.429**, whereas VisJudge achieves **MAE 0.442** and **correlation 0.681** [2510.22373].

| Model | MAE | Corr. |
|---|---:|---:|
| Claude-3.5-Sonnet | 0.823 | 0.395 |
| Claude-4-Sonnet | 0.618 | 0.470 |
| Gemini-2.0-Flash | 0.680 | 0.395 |
| Gemini-2.5-Pro | 0.661 | 0.266 |
| GPT-4o | 0.609 | 0.482 |
| GPT-5 | 0.551 | 0.429 |
| Qwen2.5-VL-7B | 1.048 | 0.322 |
| VisJudge | 0.442 | 0.681 |

Several patterns are important. First, GPT-5 has the best MAE among general MLLMs, but GPT-4o has the highest overall correlation among commercial baselines. Second, VisJudge improves over both: compared with GPT-5, MAE drops from **0.551** to **0.442**, a **19.8% reduction**, and compared with GPT-4o, correlation rises from **0.482** to **0.681**, a **41.3% improvement**. Third, performance is uneven across sub-dimensions. For GPT-5, MAEs are **0.861** for Fidelity, **0.780** for Semantic Readability, **0.776** for Insight Discovery, **0.648** for Design Style, **0.698** for Composition, and **0.682** for Color; correlations are **0.256**, **0.438**, **0.383**, **0.463**, **0.277**, and **0.295** respectively. The paper states that all models struggle significantly on **Aesthetics** dimensions, with average MAE around **0.76** and most correlations below **0.3**, except Design Style at **0.44**.

The benchmark also exposes systematic score bias. Human experts have average score **\(\mu = 3.13\)**. Several models show inflation: Qwen2.5-VL-7B at **3.89**, Claude-3.5-Sonnet at **3.87**, Gemini-2.0-Flash at **3.64**, GPT-4o at **3.53**, Claude-4-Sonnet at **3.56**, and GPT-5 at **3.36**. Gemini-2.5-Pro is described as overly conservative with **\(\mu = 3.02\)**. VisJudge nearly matches the human score distribution with **\(\mu = 3.11\)**. Complexity analysis adds a further result: performance degrades from single visualizations to multiple visualizations to dashboards, and in dashboards some baseline models even show negative Data Fidelity correlations, including Claude-3.5-Sonnet at **\(-0.031\)** and GPT-5 at **\(-0.013\)**.

## 5. VisJudge as a specialized judge model

The benchmark is not only evaluative. It is also used to train the specialized **VisJudge** model, which is built by fine-tuning **Qwen2.5-VL-7B-Instruct** on the benchmark’s human-annotated data. The model takes a visualization image together with the structured evaluation prompt based on the Fidelity, Expressiveness, and Aesthetics rubric, and it outputs a structured JSON containing per-dimension scores, rationales, and the average score [2510.22373].

Training uses **reinforcement learning with the GRPO algorithm** together with **LoRA** adaptation. The reported configuration is: **5 epochs** on **four NVIDIA A6000 GPUs (48 GB each)**, learning rate **\(10^{-5}\)**, **AdamW** optimizer, weight decay **0.01**, cosine annealing scheduler with warmup ratio **0.1**, LoRA rank and alpha **128** applied to all linear layers, **GRPO beta 0.01**, global batch size **16**, and **bfloat16** mixed precision. The software stack uses **SWIFT** with **PyTorch** and **DeepSpeed ZeRO Stage 2**.

The reward function is defined explicitly as a composite of accuracy and format rewards:

\[
R_{\text{composite}} = 0.9 R_{\text{acc}} + 0.1 R_{\text{format}}
\]

For a single score prediction, the accuracy reward is

\[
R_{\text{acc\_single}} = \exp\left(-\frac{\left|\text{score}_{\text{predicted}} - \text{score}_{\text{ground-truth}}\right|}{0.5}\right)
\]

and the final accuracy reward is the average over the six dimension scores and the overall average score. The format reward is binary: it is **1.0** if the output contains all required fields in the expected JSON structure and **0** otherwise. This training design makes structure compliance an explicit optimization target rather than only a parsing convenience.

The paper also studies VisJudge-Bench as a scaling resource. With a single training epoch and progressively larger subsets, human-model correlation rises from **0.30** to **0.65** at **2,442** samples, approximately following a logarithmic trend with **\(R^2 = 0.628\)**. Prediction error decreases smoothly: MAE drops from **1.05** to **0.45** with **\(R^2 = 0.984\)**, and MSE from **1.55** to **0.30** with **\(R^2 = 0.992\)**. The **500–1,000** sample regime yields **45%** of total correlation improvement, after which improvements continue with diminishing returns. This suggests that the benchmark functions not only as an evaluation set but also as a domain-specific training corpus for visualization judges.

## 6. Significance, limitations, and relation to judge-benchmark research

VisJudge-Bench fills a missing layer in the visualization ecosystem by benchmarking MLLMs as **visualization critics** rather than as visualization generators or chart readers. The paper identifies practical use cases including automatic visualization recommendation, chart critique, dashboard evaluation, quality control for generated visualizations, and potentially training judge models to support NL2VIS systems. Its six-dimensional labels provide both a holistic score and a structured error surface, which is useful for downstream analysis [2510.22373].

The benchmark also has clear limitations. First, judgments of visualization aesthetics and quality inevitably reflect human preferences and may encode bias. Second, the data come from web-crawled screenshots and therefore reflect the domains, tools, and publication practices visible on the web rather than a controlled sample of all visualization practices. Third, dashboards in the dataset tend to have higher quality than single or multi-view charts, which may influence model calibration. Fourth, the benchmark is built around **scalar rating prediction rather than pairwise preference learning or free-form critique alone**, although rationales are generated; the paper does not present a dedicated pairwise-comparison benchmark. These limitations matter because visualization judgment has both objective and subjective components, and the current benchmark operationalizes that mixture through expert consensus rather than through a formal theory of visual quality.

Within the broader judge-evaluation literature, VisJudge-Bench occupies a specific niche. Related work has emphasized multilingual judge reliability under **position bias**, **verbosity bias**, **order inconsistency**, and **cross-lingual degradation** [2606.22329]; adaptive, debiased, and consistent judges that model judging as a conditional policy and enforce **pointwise versus pairwise consistency** [2602.06625]; and psychometric characterization of judges through **dark current**, **stable cross-sensitivity**, **positional false preference**, **target sensitivity**, and **criterion** [2606.15610]. Other benchmarks stress contextual evidence grounding in RAG and summarization [2503.15620], video-grounded evaluation where direct access to the video matters [2509.21451], capability-oriented multimodal judge benchmarking through **pairwise CoT comparison**, **length bias avoidance**, and **process error detection** [2603.00546], and constraint-level judge evaluation with **yes/partial/no** labels plus inconsistency metrics [2605.03858]. This suggests that VisJudge-Bench is best understood as a domain-specific benchmark for **visualization-quality score prediction**, whereas much of the adjacent literature focuses more directly on judge bias, invariance, calibration, consistency, or perturbation robustness.

In that broader landscape, VisJudge-Bench establishes a benchmark and training resource for expert-aligned visualization judgment, but not yet a full reliability audit framework. Its distinctive contribution is the claim that visualization judgment requires simultaneous assessment of truthfulness, communicative value, and visual design, and that current MLLMs remain substantially below expert standards on that combined task.

Source: https://www.emergentmind.com/topics/visjudge-bench