JudgerBenchV2: Unified LLM Judge Benchmark

• JudgerBenchV2 is a comprehensive evaluation suite for LLM-as-Judge models that covers ten diverse query scenarios in both English and Chinese.
• It employs a Mixture-of-Judgers approach to establish a reliable ground truth through majority voting, ensuring robust performance measurement.
• The benchmark integrates sample-level accuracy with holistic ranking consistency, standardized pipelines, and fixed JSONL inputs for fair model comparisons.

JudgerBenchV2 is a unified, cross-domain evaluation suite specifically designed for assessing “LLM-as-Judge” models. It serves as an authoritative benchmark for measuring both judgment accuracy and model ranking consistency across a broad array of real-world user query scenarios, systematically addressing limitations observed in previous judge-model benchmarks in scope, standardization, and metric design (Zhang et al., 12 Jul 2025).

1. Benchmark Composition and Coverage

JudgerBenchV2 encompasses ten distinct user-query scenarios: summarization, translation, coding, open-domain question answering (QA), multi-turn dialogue, creative writing, style transfer, logical reasoning, factual QA, and safety-sensitive prompts. For each scenario, real-world queries in English and Chinese were sourced from CompassArena. These queries underwent clustering with K-means, after which an LLM assigned difficulty grades; from each cluster, exactly 100 queries were manually retained to ensure balanced linguistic coverage and a spectrum of challenges.

Each scenario results in 1,000 pairwise comparison instances (100 queries × 10 LLMs), with every candidate LLM response paired against a GPT-4o-mini output. This yields a corpus of 10,000 pairwise evaluations, capturing the diversity crucial for robust judge model benchmarking.

Scenario	Language Coverage	#Query Pairs/Scenario
Summarization	English, Chinese	1,000
Translation	English, Chinese	1,000
Coding	English, Chinese	1,000
Dialogue/QA/Style...	English, Chinese	1,000 each
...total (10 tasks)	English, Chinese	10,000

This broad scope distinguishes JudgerBenchV2 from previous benchmarks such as RewardBench (four categories) and JudgeBench (factual QA), providing significantly more comprehensive domain coverage.

2. Data Generation and MoJ Ground Truth

For each scenario, ten state-of-the-art LLMs (including Qwen2.5, Llama-3, DeepSeek-V3, InternLM) generate candidate responses to every query. Each output is paired with a GPT-4o-mini response, yielding systematic and exhaustive pairwise comparisons across all models and tasks.

To address subjectivity in open-ended preference assessments, JudgerBenchV2 adopts a Mixture-of-Judgers (MoJ) approach: for every response pair, three large judge models (DeepSeek-R1, DeepSeek-V3-0324, Qwen3-235B) independently determine the superior answer; the majority vote serves as the reference label. This consensus-based ground truth methodology increases labeling reliability compared to single-annotator or single-model procedures prevalent in earlier benchmarks.

The benchmark omits train/dev/test splits; all pairwise cases are presented to each judge model under evaluation. By fixing prompts and employing consistent input templates on all 10,000 JSONL entries, the suite enforces end-to-end standardization, facilitating fair and reproducible comparisons.

3. Evaluation Metrics and Scoring

JudgerBenchV2 introduces a two-component performance framework that combines sample-level judgment accuracy with model-level ranking consistency. The final metric $\mathcal{P}$ aggregates these dimensions to yield a single 0–100 score.

Sample-level Judgment Accuracy

Let $N$ denote the total number of pairwise comparisons ($N=10,000$), and $C$ the count of cases where the test judge matches the MoJ label. Then: $\mathrm{Acc} = \frac{C}{N}$

Rank Consistency and Score Difference Penalty

Define $M$ as the number of evaluated LLMs. For each model $m$:

• $s_{1,m}$, $s_{2,m}$: cumulative “wins” under MoJ and test judge, respectively
• $r_{1,m}$, $N$0: corresponding rankings (descending order)

Compute:

• average normalized rank discrepancy: $N$1
• average normalized score difference: $N$2

The final score is: $N$3

This formulation emphasizes fine-grained agreement while penalizing holistic misalignment in model rankings and win distributions. Optionally, cross-domain $N$4 can be reported as the mean over the ten scenarios, and Kendall’s $N$5 coefficient is supported for classic rank-correlation analysis: $N$6

4. Improvements over Prior Benchmarks

JudgerBenchV2 advances judge model evaluation along multiple axes:

• Scenario Breadth: Ten covered tasks substantially exceed the four of RewardBench or the factual QA focus of JudgeBench.
• Ground Truth Reliability: Majority voting among three large, expert judge models addresses the unreliability of single-label judging.
• Ranking-awareness: Integration of model-level ranking and scoring de-emphasizes mere pairwise agreement in favor of holistic evaluation.
• Pipeline Standardization: Fixed prompts and JSONL entry design ensure identical presentation across all judge models, with outputs comparably unit-scaled.

These attributes result in a benchmark that better predicts robust, cross-domain judgment capabilities in large models and isolates weaknesses masked by overly narrow or noisy assessments.

5. Empirical Baselines and Interpretive Context

Empirical results reveal the stringency of JudgerBenchV2. Off-the-shelf Llama3-8B and Qwen2.5-7B yield approximately 57% $N$7 scores. CompassJudger-1-7B reaches 57.96%, while CompassJudger-2-7B achieves 60.52%. At larger scales, CompassJudger-2-32B attains 62.21%, outperforming DeepSeek-V3-0324 (64.43%) while using substantially fewer parameters. This suggests that task-driven multi-domain curation and verifiable-reward supervision contribute substantially to generalist judge performance (Zhang et al., 12 Jul 2025).

A plausible implication is that as LLMs-as-judge models become more central to evaluation pipelines, benchmarks like JudgerBenchV2 will be foundational both for model selection and for analyzing failure modes in critical, high-variance application landscapes.

6. Standardized Usage Protocol

JudgerBenchV2 provides an open-source, reproducible evaluation protocol:

1. Repository Access: git clone https://github.com/open-compass/CompassJudger && cd CompassJudger/judgerbenchv2
2. Dependencies: pip install -r requirements.txt (Python ≥3.10)
3. Run Judgments: $N=10,000$0
4. Metric Computation:

$N=10,000$1 This script reports sample-level accuracy, normalized rank/score discrepancies, and the overall $N$8.

1. Comparative Analysis: Baseline scores and per-scenario $N$9 breakdowns are supported for diagnosing model strengths and weaknesses.

By enforcing a single, unified protocol with consistent task distribution and evaluation mathematics, JudgerBenchV2 enables robust, transparent tracking of progress in LLM-as-judge development and application.

7. Significance and Prospects

JudgerBenchV2 establishes a new evaluative standard for generalist judge models, directly addressing the need for rigorous, standardized, and broad-scope model assessment. The adoption of a Mixture-of-Judgers label, the integration of both accuracy and rank consistency, and the multi-domain structure collectively represent a significant methodological advance. As model assessment moves toward greater automation and broader applicability, standardized frameworks such as JudgerBenchV2 are poised to play a central role in both benchmarking and diagnostic research (Zhang et al., 12 Jul 2025).