VisJudge: MLLM Judge for Visualization Quality

• VisJudge is a domain-specific model that automatically assesses chart quality through its threefold focus on Fidelity, Expressiveness, and Aesthetics.
• The system employs a fine-tuned multimodal LLM with reinforcement learning to align its ratings closely with expert human judgments.
• VisJudge-Bench offers a comprehensive benchmark of 3,090 curated real-world visualizations to rigorously test model performance across multiple chart types.

Searching arXiv for the specified paper and closely related work on VisJudge and LLM/VLM-as-a-judge. VisJudge is a specialized “MLLM-as-a-judge” system for data visualization, and VisJudge-Bench is the benchmark that makes it possible to build and test such a system rigorously. Together they address whether multimodal LLMs can reliably judge whether a chart or dashboard is accurate, informative, and well-designed in the same way that an expert data visualization practitioner would. The benchmark contains 3,090 expert-annotated samples from real-world scenarios across single visualizations, multiple visualizations, and dashboards, and the model is a fine-tuned multimodal LLM designed for six-dimensional visualization quality assessment (Xie et al., 25 Oct 2025).

1. Problem scope and conceptual definition

VisJudge addresses automatic assessment of visualization quality. In this setting, quality is explicitly threefold: Fidelity, Expressiveness, and Aesthetics. Fidelity concerns whether data are encoded truthfully, including axes, baselines, scales, area or length encoding, 3D effects, and distortions. Expressiveness concerns whether a visualization clearly communicates and supports analysis, including Semantic Readability and Insight Discovery. Aesthetics concerns whether the visual design is professionally executed and pleasing, including Design Style, Visual Composition, and Color Harmony (Xie et al., 25 Oct 2025).

A recurrent misconception is to equate visualization evaluation with natural-image aesthetic judgment. VisJudge is defined against that reduction. The motivating argument is that a good visualization must satisfy all three pillars—if it lies with the data, is unreadable, or is ugly and cluttered, it fails. This differentiates visualization assessment from natural-image aesthetic datasets such as AVA and ArtiMuse, which focus almost entirely on aesthetic appeal of photos and artworks, and from chart QA benchmarks such as ChartQA, PlotQA, and ChartInsights, which test reading and reasoning about charts rather than how well the chart is designed (Xie et al., 25 Oct 2025).

The system also differs from benchmarks centered on NL2VIS correctness, such as VisEval and VIS-Shepherd. Those works evaluate whether a generated chart matches a query, whereas VisJudge evaluates the chart’s intrinsic quality across a structured, multidimensional rubric. Related chart-judge studies further showed that open-source LVLM judges can be strong but variable, with some achieving about 80% agreement with GPT-4 judgments and others falling below about 10%, while positional preference and length bias persist (Laskar et al., 13 May 2025). This broader context helps explain why VisJudge is framed not as a generic multimodal evaluator, but as a domain-specific judge for visualization quality (Xie et al., 25 Oct 2025).

2. VisJudge-Bench: corpus, taxonomy, and benchmark structure

VisJudge-Bench is presented as the first comprehensive benchmark for evaluating MLLMs’ performance in assessing visualization aesthetics and quality. It contains 3,090 visualization instances collected from real-world images crawled from the web via Bing Image Search using over 2,000 keywords spanning chart types, domains, and quality modifiers. The collection pipeline reduced an initial pool of more than 300,000 images to 80,210 via heuristics, then to 13,220 via GPT-4o classification and human checks, and finally to 3,090 samples via stratified sampling (Xie et al., 25 Oct 2025).

The corpus is balanced across three high-level categories.

Category	Samples	Share
Single visualizations	1,041	33.7%
Multiple visualizations	1,024	33.1%
Dashboards	1,025	33.2%

The taxonomy spans 32 chart and view types. The single-visualization category contains 22 subtypes, including Bar Chart, Line Chart, Area Chart, Pie Chart, Donut Chart, Histogram, Scatter Plot, Box Plot, Funnel Chart, Choropleth Map, Point Map, Treemap, Sankey Diagram, Bubble Chart, Radar Chart, Network Graph, Candlestick Chart, Gauge Chart, Word Cloud, Violin Plot, and Other Single View. Multiple visualizations contain five subtypes: Comparison Views, Small Multiples, Coordinated Views, Overview–Detail, and Other Multi View. Dashboards contain five subtypes: Analytical Dashboard, Operational Dashboard, Interactive Dashboard, Strategic Dashboard, and Other Dashboard (Xie et al., 25 Oct 2025).

Each sample is one visualization instance represented by a single image file. It may be a single chart, a multi-view composite, or a full dashboard with multiple charts, KPI tiles, filters, and related components. Evaluation is performed on the image as-is; no underlying data tables are required for benchmark use. This design makes the benchmark directly applicable to MLLM evaluation, while also imposing a specific limitation: data-level fidelity is inferred from visual evidence rather than reconstructed from raw data (Xie et al., 25 Oct 2025).

The dataset is split into 2,163 training samples, 279 validation samples, and 648 test samples, stratified by visualization type so that the test set mirrors the full dataset in type and quality distributions. The benchmark task is framed as regression: given a visualization image, predict six integer scores from 1 to 5 plus an overall average score aligned with human judgments (Xie et al., 25 Oct 2025).

3. Fidelity–Expressiveness–Aesthetics framework and annotation protocol

VisJudge-Bench operationalizes visualization quality into six numeric dimensions on a 1–5 scale. Under Fidelity, the benchmark uses Data Fidelity. Under Expressiveness, it uses Semantic Readability and Insight Discovery. Under Aesthetics, it uses Design Style, Visual Composition, and Color Harmony. Each sample therefore has seven labels: the six dimension scores plus an overall score defined as the average of the six (Xie et al., 25 Oct 2025).

The dimension semantics are precise. Data Fidelity measures the faithfulness of encodings to underlying data, including axes, baselines, scales, aspect ratio, 3D distortions, and cropping. Semantic Readability measures clarity of the meaning of visual elements such as marks, colors, legends, and labels. Insight Discovery measures ease of seeing meaningful patterns such as trends, outliers, clusters, and business-relevant insights. Design Style evaluates innovation, uniqueness, and professional polish. Visual Composition evaluates layout, alignment, spacing, size ratios, and information density. Color Harmony evaluates coherence of palette, contrast, number of colors, and saturation (Xie et al., 25 Oct 2025).

The rubrics are adapted to visualization structure. For example, the description of Data Fidelity differs between single charts and dashboards because cross-view consistency matters in multi-view and dashboard settings. For a single chart, a score of $1$ includes severe misrepresentation such as a misleading 3D pie or a non-zero bar baseline exaggerating differences; a score of $5$ includes appropriate baselines, justified scales, consistent labels and encodings, and no deformation (Xie et al., 25 Oct 2025).

The overall score distribution is reported as roughly normal, with mean approximately 3.13, standard deviation approximately 0.72, and range 1.00–4.89. Dashboards skew somewhat higher, with mean approximately 3.56, while single and multi-view charts are both approximately 2.91–2.92. This suggests that dashboards in the collected corpus are relatively curated, although that interpretation remains descriptive rather than causal (Xie et al., 25 Oct 2025).

Annotation relied on crowdsourced experts recruited via CloudResearch under strict filters: at least a Bachelor’s degree, 97–100% approval rate on past tasks, 100–10,000 prior tasks, native English speakers, occupations in business, finance, STEM, design, and related fields, age 20–50, and recent platform activity. Annotators were paid USD \$10/hour. Each task contained 15 visualizations—5 single, 5 multi, and 5 dashboards—and each visualization required six ratings, for 90 questions per batch. Every sample was annotated by three independent annotators (Xie et al., 25 Oct 2025).

Quality control combined embedded validation checks, statistical conflict detection, and expert adjudication. High-disagreement samples were flagged using per-sample standard deviation greater than 1.0, with additional heuristics for outlier removal, malicious scoring patterns, and dimension-specific deviations greater than 2.0 points. A team of three visualization experts reviewed these cases through a custom interface. Among the 3,090 samples, 2,606 had alternative candidate results and 1,792 benefited from expert calibration. The output is described as a gold-standard human score per dimension and per sample (Xie et al., 25 Oct 2025).

4. Model architecture, prompting, and reinforcement-learning objective

VisJudge is a fine-tuned multimodal LLM designed specifically to judge visualization quality under the six-dimension framework. Its base model is Qwen2.5-VL-7B-Instruct, described as a 7B-parameter vision-LLM with a ViT-like visual backbone and Transformer text backbone. Visualizations are processed through Qwen2.5-VL’s visual encoder, which produces image embeddings, while textual elements such as titles, labels, legends, and axis text are read directly by the underlying vision-LLM rather than by a separate OCR or chart-specific parser (Xie et al., 25 Oct 2025).

At inference time, VisJudge takes the visualization image and a structured evaluation prompt based on the Fidelity–Expressiveness–Aesthetics framework. The prompt describes the chart type, specifies the meaning of each dimension, and asks for a numeric score, a short textual reasoning, and an overall average score. The expected output is a JSON object containing the six dimension fields and average_score. The structured format is not incidental: it is used directly in the reward design during training (Xie et al., 25 Oct 2025).

Training uses reinforcement learning over Qwen2.5-VL with GRPO (Group Relative Policy Optimization) and LoRA. LoRA is applied to linear layers with rank 128. Training runs for 5 epochs with learning rate $10^{-5}$, AdamW optimizer, cosine schedule, weight decay 0.01, and bfloat16 mixed precision on four NVIDIA A6000 GPUs. The training split of VisJudge-Bench provides 2,163 samples; validation and test remain held out (Xie et al., 25 Oct 2025).

The reward is a composite of score accuracy and output formatting:

$$R_{\text{composite}} = 0.9\,R_{\text{acc}} + 0.1\,R_{\text{format}}.$$

For each score field, the accuracy component uses

$$R_{\text{acc\_single}} = \exp\left( - \frac{\left|\text{score}_{\text{predicted}} - \text{score}_{\text{ground-truth}}\right|}{0.5} \right).$$

The overall accuracy reward is the average of the per-field rewards. The format reward is binary: $R_{\text{format}} = 1$ if the output JSON is complete and $R_{\text{format}} = 0$ otherwise (Xie et al., 25 Oct 2025).

This reward design trains both the content and the packaging of the judgment. A plausible implication is that the model is optimized not only to approximate human scores but also to behave as a reliable structured evaluator in downstream pipelines. That emphasis aligns VisJudge with broader judge-model work that treats evaluators as structured systems rather than single-prompt scorers, including modular judge-time compute in Verdict (Kalra et al., 25 Feb 2025) and policy-level consistency optimization in FairJudge (Yang et al., 6 Feb 2026).

5. Empirical performance, error patterns, and comparison with generic MLLMs

On the 648-sample test set, VisJudge is compared against GPT-5, GPT-4o, Claude-4-Sonnet, Claude-3.5-Sonnet, Gemini-2.0-Flash, Gemini-2.5-Pro, and the untuned Qwen2.5-VL-7B-Instruct base model. All models are prompted with the same evaluation prompt, run three times per sample, and averaged to reduce randomness. Temperature is 0.8 for VisJudge and other open models via vLLM (Xie et al., 25 Oct 2025).

The headline overall result is that VisJudge reduces the gap to human experts relative to generic MLLMs. GPT-5 obtains overall MAE 0.551, MSE 0.484, and correlation 0.429. GPT-4o attains MAE 0.609 and the best baseline correlation at 0.482. VisJudge attains overall MAE 0.442, MSE 0.306, and correlation 0.681. The paper reports this as a 19.8% MAE reduction versus GPT-5 and a 58.7% improvement in consistency with human experts relative to GPT-5’s correlation (Xie et al., 25 Oct 2025).

The six per-dimension MAEs for VisJudge are 0.662 for Data Fidelity, 0.649 for Semantic Readability, 0.679 for Insight Discovery, 0.581 for Design Style, 0.546 for Visual Composition, and 0.604 for Color Harmony. Per-dimension correlations are 0.571, 0.625, 0.572, 0.567, 0.512, and 0.385 respectively. The weakest dimension remains Color Harmony, which the paper associates with subjectivity and possible cultural bias; dashboards also remain the hardest structural category, with correlation approximately 0.375 compared with approximately 0.577 for single visualizations and approximately 0.565 for multiple visualizations (Xie et al., 25 Oct 2025).

The model’s score distribution is another notable result. Human overall scores have mean $\mu = 3.13$. Several generic MLLMs exhibit inflation or concentration: Qwen2.5-VL-7B has $\mu = 3.89$, Claude-3.5-Sonnet has $\mu = 3.87$, GPT-4o has $5$0, GPT-5 has $5$1, and Gemini-2.5-Pro has $5$2. VisJudge’s score distribution has mean $5$3, described as extremely close to the human mean and more spread and balanced than the baselines. This suggests that part of VisJudge’s gain is calibration, not only ranking (Xie et al., 25 Oct 2025).

Qualitative cases illustrate typical failure modes of generic MLLMs. In a chaotic treemap with human average rating 1.67, Qwen2.5-VL-7B assigns 3.67 and Claude-4-Sonnet 3.08, both over-praising the chart, whereas VisJudge assigns 2.00 and identifies chaotic layout, difficult interpretation of sizes and categories, and poor composition. In another case, Gemini-2.5-Pro exhibits a conservative bias on a high-quality dashboard, assigning 2.94 against a human rating of 4.17, while VisJudge assigns 3.83. These cases are consistent with the broader observation that generic MLLMs often over-praise low-quality charts or over-penalize high-quality but slightly unconventional ones (Xie et al., 25 Oct 2025).

A second misconception is therefore that “human-like distribution” implies expert parity. The reported numbers do not support that conclusion. Even after specialization, VisJudge remains far from perfect on dashboards and on some aesthetic dimensions. Related studies on hidden shortcuts in LLM-based evaluation further suggest that large verdict shifts can coexist with near-zero cue acknowledgment, which raises a general caution about explanation faithfulness in judge models (Marioriyad et al., 8 Feb 2026).

6. Applications, related judge paradigms, and limitations

VisJudge has several explicit applications. The paper identifies automated visualization review tools, real-time design assistants in BI systems such as Power BI and Tableau, educational tools for teaching visualization practice, and critic modules for evaluating charts generated by NL2VIS systems. Because VisJudge outputs both scores and textual reasoning per dimension, its judgments are presented as interpretable and actionable; examples include identifying a non-zero baseline that exaggerates differences or a color palette with too many saturated hues that causes visual fatigue (Xie et al., 25 Oct 2025).

Within the broader judge literature, VisJudge occupies one point in a larger design space. One axis is domain specialization: VisJudge is specialized for visualization quality, whereas general chart-judge work evaluated open LVLMs as judges for chart comprehension and reasoning and found substantial model-to-model variability as well as persistent positional preference and length bias (Laskar et al., 13 May 2025). Another axis is post-hoc calibration: quantitative LLM judges use a frozen base judge plus a small regression-type model to align judge scores with human scores, a strategy described as more computationally efficient than supervised fine-tuning and potentially more statistically efficient when human feedback is limited (Sahoo et al., 3 Jun 2025). A plausible implication is that VisJudge-like systems could in principle be further calibrated using post-hoc quantitative layers, although that is not part of the reported VisJudge architecture.

A further axis concerns inference-time structure and adaptivity. Verdict treats judging as a pipeline of modular reasoning units such as verification, debate, and aggregation, scaling judge-time compute instead of relying on a single monolithic evaluation call (Kalra et al., 25 Feb 2025). Jury-on-Demand selects a per-instance jury of judges using learned reliability predictors and reliability-weighted aggregation, rather than a static committee (Li et al., 1 Dec 2025). FairJudge frames judging as a learnable and regularized policy, explicitly targeting adaptivity, debiasing, and cross-mode consistency (Yang et al., 6 Feb 2026). These works do not replace VisJudge’s domain-specific contribution, but they clarify that visualization judging can be analyzed not only as a benchmark-and-model problem but also as a systems problem involving calibration, bias control, and evaluation consistency.

The main limitations stated for VisJudge-Bench and VisJudge are threefold. First, the data are sourced from web images and heavily filtered, so they may overrepresent certain styles, tools, or Western aesthetics; color and style evaluation may therefore be culturally biased. Second, the benchmark operates on static images without access to underlying data tables or interactive behaviors, so data fidelity is approximated through visual cues and cannot capture hidden data-level errors. Third, dashboards remain difficult, with substantially lower correlation than single charts or multi-view compositions. Future directions proposed in the paper include fairness and bias analysis across visualization styles or cultural contexts, data-aware evaluation when raw data are available, extension to interactive and narrative visualizations, and using VisJudge-Bench to train generative systems that satisfy quality criteria rather than only syntactic correctness (Xie et al., 25 Oct 2025).

In that sense, VisJudge is best understood as a domain-specific judge model and benchmark that formalize visualization assessment as a multidimensional machine-learning problem. Its central contribution is not merely higher agreement with human experts, but the explicit operationalization of visualization quality into Fidelity, Expressiveness, and Aesthetics, together with a training and evaluation protocol that makes those dimensions measurable, reproducible, and extensible within the wider literature on multimodal judges (Xie et al., 25 Oct 2025).