---
title: 'JudgerBenchV2: Unified LLM Judge Benchmark'
url: https://www.emergentmind.com/topics/judgerbenchv2
type: topic
---

# JudgerBenchV2: Unified LLM Judge Benchmark

JudgerBenchV2 is a unified, cross-domain evaluation suite specifically designed for assessing “LLM-as-Judge” models. It serves as an authoritative benchmark for measuring both judgment accuracy and model ranking consistency across a broad array of real-world user query scenarios, systematically addressing limitations observed in previous judge-model benchmarks in scope, standardization, and metric design [2507.09104].

## 1. Benchmark Composition and Coverage

JudgerBenchV2 encompasses ten distinct user-query scenarios: summarization, translation, coding, open-domain question answering (QA), multi-turn dialogue, creative writing, style transfer, logical reasoning, factual QA, and safety-sensitive prompts. For each scenario, real-world queries in English and Chinese were sourced from CompassArena. These queries underwent clustering with K-means, after which an LLM assigned difficulty grades; from each cluster, exactly 100 queries were manually retained to ensure balanced linguistic coverage and a spectrum of challenges.

Each scenario results in 1,000 pairwise comparison instances (100 queries × 10 LLMs), with every candidate LLM response paired against a GPT-4o-mini output. This yields a corpus of 10,000 pairwise evaluations, capturing the diversity crucial for robust judge model benchmarking.

| Scenario             | Language Coverage | #Query Pairs/Scenario |
|----------------------|------------------|----------------------|
| Summarization        | English, Chinese | 1,000                |
| Translation          | English, Chinese | 1,000                |
| Coding               | English, Chinese | 1,000                |
| Dialogue/QA/Style... | English, Chinese | 1,000 each           |
| ...total (10 tasks)  | English, Chinese | 10,000               |

This broad scope distinguishes JudgerBenchV2 from previous benchmarks such as RewardBench (four categories) and JudgeBench (factual QA), providing significantly more comprehensive domain coverage.

## 2. Data Generation and MoJ Ground Truth

For each scenario, ten state-of-the-art LLMs (including Qwen2.5, Llama-3, DeepSeek-V3, InternLM) generate candidate responses to every query. Each output is paired with a GPT-4o-mini response, yielding systematic and exhaustive pairwise comparisons across all models and tasks.

To address subjectivity in open-ended preference assessments, JudgerBenchV2 adopts a Mixture-of-Judgers (MoJ) approach: for every response pair, three large judge models (DeepSeek-R1, DeepSeek-V3-0324, Qwen3-235B) independently determine the superior answer; the majority vote serves as the reference label. This consensus-based ground truth methodology increases labeling reliability compared to single-annotator or single-model procedures prevalent in earlier benchmarks.

The benchmark omits train/dev/test splits; all pairwise cases are presented to each judge model under evaluation. By fixing prompts and employing consistent input templates on all 10,000 JSONL entries, the suite enforces end-to-end standardization, facilitating fair and reproducible comparisons.

## 3. Evaluation Metrics and Scoring

JudgerBenchV2 introduces a two-component performance framework that combines sample-level judgment accuracy with model-level ranking consistency. The final metric $\mathcal{P}$ aggregates these dimensions to yield a single 0–100 score.

### Sample-level Judgment Accuracy

Let $N$ denote the total number of pairwise comparisons ($N=10,000$), and $C$ the count of cases where the test judge matches the MoJ label. Then:
\[
\mathrm{Acc} = \frac{C}{N}
\]

### Rank Consistency and Score Difference Penalty

Define $M$ as the number of evaluated LLMs. For each model $m$:
- $s_{1,m}$, $s_{2,m}$: cumulative “wins” under MoJ and test judge, respectively
- $r_{1,m}$, $r_{2,m}$: corresponding rankings (descending order)

Compute:
- average normalized rank discrepancy:
\[
\frac{1}{M}\sum_{m=1}^{M}\frac{|r_{1,m}-r_{2,m}|}{M-1}
\]
- average normalized score difference:
\[
\frac{1}{M}\sum_{m=1}^{M}\frac{|s_{1,m}-s_{2,m}|}{\max_{m'}|s_{1,m'}-s_{2,m'}|}
\]

The final score is:
\[
\mathcal{P} = 100\frac{C}{N} - 100\left[
\frac{1}{M}\sum_{m}\frac{|r_{1,m}-r_{2,m}|}{M-1}
+ \frac{1}{M}\sum_{m}\frac{|s_{1,m}-s_{2,m}|}{\max_{m'}|s_{1,m'}-s_{2,m'}|}
\right]
\]

This formulation emphasizes fine-grained agreement while penalizing holistic misalignment in model rankings and win distributions. Optionally, cross-domain $\mathrm{Acc}$ can be reported as the mean over the ten scenarios, and Kendall’s $\tau$ coefficient is supported for classic rank-correlation analysis:
\[
\tau = \frac{2}{M(M-1)}\sum_{i<j}\mathrm{sgn}[(r_{1,i}-r_{1,j})(r_{2,i}-r_{2,j})]
\]

## 4. Improvements over Prior Benchmarks

JudgerBenchV2 advances judge model evaluation along multiple axes:

- **Scenario Breadth:** Ten covered tasks substantially exceed the four of RewardBench or the factual QA focus of JudgeBench.
- **Ground Truth Reliability:** Majority voting among three large, expert judge models addresses the unreliability of single-label judging.
- **Ranking-awareness:** Integration of model-level ranking and scoring de-emphasizes mere pairwise agreement in favor of holistic evaluation.
- **Pipeline Standardization:** Fixed prompts and JSONL entry design ensure identical presentation across all judge models, with outputs comparably unit-scaled.

These attributes result in a benchmark that better predicts robust, cross-domain judgment capabilities in large models and isolates weaknesses masked by overly narrow or noisy assessments.

## 5. Empirical Baselines and Interpretive Context

Empirical results reveal the stringency of JudgerBenchV2. Off-the-shelf Llama3-8B and Qwen2.5-7B yield approximately 57% $\mathcal{P}$ scores. CompassJudger-1-7B reaches 57.96%, while CompassJudger-2-7B achieves 60.52%. At larger scales, CompassJudger-2-32B attains 62.21%, outperforming DeepSeek-V3-0324 (64.43%) while using substantially fewer parameters. This suggests that task-driven multi-domain curation and verifiable-reward supervision contribute substantially to generalist judge performance [2507.09104].

A plausible implication is that as LLMs-as-judge models become more central to evaluation pipelines, benchmarks like JudgerBenchV2 will be foundational both for model selection and for analyzing failure modes in critical, high-variance application landscapes.

## 6. Standardized Usage Protocol

JudgerBenchV2 provides an open-source, reproducible evaluation protocol:

1. **Repository Access:**  
   `git clone https://github.com/open-compass/CompassJudger && cd CompassJudger/judgerbenchv2`

2. **Dependencies:**  
   `pip install -r requirements.txt` (Python ≥3.10)

3. **Run Judgments:**  
   ```bash
   python run_judgerbenchv2.py --input judgerbenchv2.jsonl --model your_judge_model --output your_preds.jsonl
   ```

4. **Metric Computation:**  
   ```bash
   python compute_judgerbenchv2.py --gt judgerbenchv2_gt.jsonl --pred your_preds.jsonl
   ```
   This script reports sample-level accuracy, normalized rank/score discrepancies, and the overall $\mathcal{P}$.

5. **Comparative Analysis:**  
   Baseline scores and per-scenario $\mathrm{Acc}_d$ breakdowns are supported for diagnosing model strengths and weaknesses.

By enforcing a single, unified protocol with consistent task distribution and evaluation mathematics, JudgerBenchV2 enables robust, transparent tracking of progress in LLM-as-judge development and application.

## 7. Significance and Prospects

JudgerBenchV2 establishes a new evaluative standard for generalist judge models, directly addressing the need for rigorous, standardized, and broad-scope model assessment. The adoption of a Mixture-of-Judgers label, the integration of both accuracy and rank consistency, and the multi-domain structure collectively represent a significant methodological advance. As model assessment moves toward greater automation and broader applicability, standardized frameworks such as JudgerBenchV2 are poised to play a central role in both benchmarking and diagnostic research [2507.09104].

Source: https://www.emergentmind.com/topics/judgerbenchv2