---
title: 'LongJudgeBench: Evaluating Long-Form LLM Judges'
url: https://www.emergentmind.com/topics/longjudgebench
type: topic
---

# LongJudgeBench: Evaluating Long-Form LLM Judges

Searching arXiv for LongJudgeBench and closely related judge-benchmark papers.
LongJudgeBench is a benchmark for **meta-evaluating LLM-as-a-judge systems on long-form outputs** across diverse scenarios and judgment protocols. It was introduced to address a gap in prior judge benchmarks, which the paper characterizes as being focused mainly on short-form outputs: benchmarks such as MT-Bench, LLMBar, RewardBench, PreferenceBench, RPR, JudgeBench, and RewardBench 2 mostly contain outputs only a few hundred tokens long, whereas LongJudgeBench’s candidate outputs average **9,249.7 tokens**. The benchmark’s central claim is that long-form evaluation is not merely a matter of output length; it requires document-level assessment of overall organization, task-relevant coverage and depth, cross-section consistency, presentation quality, and scenario-specific quality criteria. LongJudgeBench therefore evaluates whether current LLM judges can reliably assess realistic long-form generations under expert human reference judgments rather than LLM-generated pseudo-gold labels [2606.01629].

## 1. Conceptual basis and formalization

LongJudgeBench treats each evaluation instance as
\[
x_i = (\mathcal{I}_i, \mathcal{C}_i, z_i^{*}),
\]
where \(\mathcal{I}_i\) is the instruction, \(\mathcal{C}_i\) is the content to evaluate, and \(z_i^{*}\) is the reference judgment. The instruction is formalized as
\[
\mathcal{I} \doteq \{q, m\} \cup \mathcal{A},
\]
with \(q\) the original generation query or task, \(m\) the judging protocol, and \(\mathcal{A}\) optional auxiliary information such as rubrics, weights, references, domain constraints, or formatting requirements. The content depends on the protocol:
\[
\mathcal{C} =
\begin{cases}
\{y_1\}, & \text{pointwise scoring},\\
\{y_1, y_2\}, & \text{pairwise comparison},\\
\{y_1, \ldots, y_K\}, & \text{listwise ranking}.
\end{cases}
\]
A judge \(J_\theta\) produces
\[
\hat{R}_i = J_{\theta}(\mathcal{I}_i, \mathcal{C}_i),
\]
and the response may contain both explanation and final judgment,
\[
\hat{R}_i = (\hat{e}_i, \hat{z}_i).
\]
This formulation makes the benchmark explicitly multi-protocol: it covers pointwise scoring, pairwise comparison, and listwise ranking within a single meta-evaluation framework [2606.01629].

The benchmark is motivated by the claim that long-form evaluation differs qualitatively from short-form evaluation. The issue is not only that outputs are longer, but that judges must assess holistic document properties such as structure, depth, consistency across sections, and domain-specific criteria. The paper also positions LongJudgeBench against concurrent work such as Long-form RewardBench, arguing that such resources are less suitable for meta-evaluating judges because many gold labels are produced through an LLM-judge pipeline rather than expert humans, and because the responses are shorter, around 3K tokens rather than LongJudgeBench’s 9K-plus average. This suggests that LongJudgeBench is designed as a benchmark for **judge reliability under realistic long-form conditions**, rather than merely as a longer version of short-answer judge evaluation [2606.01629].

## 2. Scope, scenarios, and dataset composition

LongJudgeBench spans **five scenarios** and **six datasets**, across **Chinese and English**, with **1,944 instances**, **1,966 outputs**, and an **average output length of 9,249.7 tokens**. The scenarios are deep research, scientific survey, creative writing, long-chain analysis, and systematic review [2606.01629].

The **deep research** scenario contains two datasets. **DR-Bench** is Chinese, with **200 instances / 200 outputs**, uses **pointwise** evaluation, and has mean output length **18,012.9 tokens** with median **9,604.5**. **RealDR (RealDeepResearch)** is Chinese and English, with **640 instances / 640 outputs**, also uses **pointwise** evaluation, and has mean output length **10,131.4** with median **9,610.0**. This scenario targets long research reports with coverage, insight, structure, and realistic document features.

The **scientific survey** scenario uses **SurGE**, an English dataset with **82 instances / 164 outputs**, evaluated **listwise**, with mean output length **28,758.0** and median **35,948.5**. It tests whether judges can recover human rankings over scientific surveys for **structure** and **content**. The **creative writing** scenario uses a **WritingPreferenceBench** subset, denoted **WP-Bench**, with **526 instances / 526 outputs**, evaluated **pairwise**, and mean output length **3,508.9** with median **2,178.5**. The **long-chain analysis** scenario uses **Verify**, with **316 instances / 316 outputs**, evaluated **pointwise**, and mean output length **3,052.6** with median **1,692.5**. The **systematic review** scenario uses **MA**, with **180 instances / 120 outputs**, evaluated **pairwise**, and mean output length **4,764.1** with median **4,515.5** [2606.01629].

These datasets cover domains including science and technology, finance, software, art, history, education, medicine, sociology, law, philosophy, environmental engineering, creative genres, and clinical medicine. A notable design choice is that LongJudgeBench preserves heterogeneous original protocols whenever possible and then normalizes them into a unified meta-evaluation setup. This suggests that the benchmark is intended to capture **real long-form evaluation settings** rather than a single synthetic task family [2606.01629].

## 3. Human reference judgments and scenario-specific supervision

A major feature of LongJudgeBench is its reliance on **expert human judgments**. In **DR-Bench**, the benchmark uses the open-source Chinese subset of DeepResearch Bench, consisting of **50 Chinese research tasks** and **200 reports** from four deep research agents: **openai-deepresearch**, **gemini-2.5-pro-deepresearch**, **grok-deeper-search**, and **perplexity-Research**. Human scoring is provided by **3 experts** with relevant research backgrounds, who rate each report on **0–100** over four dimensions: **comprehensiveness**, **insight**, **instruction following**, and **readability**. The final score is a weighted sum of these dimensions, and the original benchmark reports **68.44% inter-annotator agreement** [2606.01629].

**RealDeepResearch** is the main newly constructed dataset. It contains **40 tasks**—**29 Chinese** and **11 English**—across **12 disciplines**, and each task is paired with **4 prompt styles** and **4 generation models** (**Gemini, Grok, Mita, Perplexity**), yielding **640 documents**. Each document is scored by **2 annotators** with relevant backgrounds on **0–10** across **logical structure**, **presentation form**, and **bias checking**; if disagreement is large, a **third annotator adjudicates**. The reported agreement is
\[
ICC_{avg} = 0.76,
\]
which the authors interpret as high agreement. Total annotation cost was approximately **USD 2,260** [2606.01629].

In **SurGE**, **41 topics** in computer science each have **4 surveys**—**GT**, **AutoSurvey**, **ID**, and **Naive**—ranked by **4 PhD students in computer science**, supervised by **1 faculty member**. Each survey is independently annotated by **2 experts** on a **1–4 scale** for **structural quality** and **content quality**, with agreement
\[
\kappa = 0.792.
\]
In **WP-Bench**, the benchmark samples **263 human-validated preference pairs**, doubled to **526 pairwise instances** by swapping candidate order. Human annotation is performed by **11 professional writing evaluators** who underwent an **8-hour rubric calibration process** and score creativity, style, and emotional resonance on **0–3**. Pairs require at least **2 of 3 annotators** to agree on preference, ties are excluded, and the minimum score gap is
\[
\Delta \geq 1.
\]
In **VerifyBench-Hard**, LongJudgeBench samples **316 instances** with at least **2 annotators** per instance and reported inter-annotator agreement **0.88–0.92**. In **MA**, the benchmark uses **30 papers** from clinical systematic review and meta-analysis data, compares three fixed system pairings, and obtains **180 pairwise evaluation instances** after order swapping; judgments come from **8 annotators with medical backgrounds** using a five-level preference scale **A2, A1, tie, B1, B2** [2606.01629].

This supervision design matters because it anchors LongJudgeBench in **expert human reference judgments** across both objective and subjective scenarios. A plausible implication is that the benchmark is intended not only to compare judges, but also to expose where long-form evaluation remains dependent on domain expertise and protocol design.

## 4. Judge models, prompting settings, and metrics

The benchmark evaluates **Qwen3-32B (without thinking)**, **Qwen3-32B (with thinking)**, **Qwen3-Max**, **GPT-4o-mini**, **GPT-5.2**, **DeepSeek-V4-Flash**, **GLM-5.1**, and **Kimi-K2.6**. For each base model, four judging settings are tested: **Vanilla**, **Rubric**, **Reference**, and **Reference + Rubric**. All runs use the same task inputs, the same extraction procedure, and **temperature = 0** for fairness and determinism [2606.01629].

The benchmark compares LLM judgments against expert human references under pointwise, pairwise, and listwise protocols. To make comparison consistent, it derives pairwise preferences where needed: directly from pairwise judgments, by comparing scalar scores for pointwise tasks, and by comparing ranks for listwise tasks. For pairwise tasks, candidate order is explicitly swapped and results are averaged to reduce position bias [2606.01629].

The primary metric is **accuracy**. For preference-based datasets, if \(\mathcal{P}_q\) is the set of non-tie human-labeled candidate pairs for query \(q\), then
\[
\mathrm{Acc}_q = \frac{1}{|\mathcal{P}_q|} \sum_{(a,b)\in\mathcal{P}_q}\mathbb{I}[\hat{z}_{q,a,b}=z^{*}_{q,a,b}],
\]
where tie predictions are counted as incorrect if the human label is non-tie. The benchmark also reports **Spearman’s rank correlation** and **Kendall’s \(\tau_b\)**. Given reference and predicted score vectors, Spearman correlation is computed from their induced ranks, and Kendall’s \(\tau_b\) is
\[
\tau_{b,q} = \frac{C_q - D_q}{\sqrt{(C_q + D_q + T_{\hat{s},q})(C_q + D_q + T_{s^*,q})}},
\]
with \(C_q\) and \(D_q\) the numbers of concordant and discordant pairs, and \(T_{\hat{s},q}\), \(T_{s^*,q}\) ties occurring only in predicted or reference scores. The paper states that accuracy is the primary metric, while Spearman and Kendall are auxiliary [2606.01629].

## 5. Empirical results and scenario-specific behavior

LongJudgeBench’s headline result is that current LLM judges remain only **moderately reliable** on long-form output evaluation. Across all **32 model-setting combinations**, only **12 exceed 0.60** average accuracy, and the **overall mean is 0.5627**. The best overall configuration is **Qwen3-Max + Reference** with average accuracy **0.6721**, followed by **DeepSeek-V4-Flash + Reference** at **0.6626**, and **GLM-5.1 + Reference** at **0.6485** [2606.01629].

Performance varies sharply by scenario. **Verify** is the easiest scenario, with average performance around **0.6742**, and **DeepSeek-V4-Flash + Reference** reaches **0.8924**. **DR-Bench** is also relatively strong, with average around **0.6454**; rubric-based settings help substantially here, with **DeepSeek-V4-Flash + Rubric** and **Kimi-K2.6 + Rubric** both reaching **0.7200**. By contrast, **RealDR** is much harder, with average around **0.4393** and best scores only in the high 0.5s; **DeepSeek-V4-Flash + Rubric** reaches **0.5710**, and **GLM-5.1 + Rubric** reaches **0.5606**. **MA** is also difficult, with average around **0.4983**, while **WP-Bench** averages around **0.5776**. **SurGE** is mixed, with some judges performing well on structure or content, but results remaining unstable across dimensions and settings [2606.01629].

The most consistent aid is **Reference** information. Overall average accuracy improves from **0.5313** in **Vanilla** to **0.5843** in **Reference**. On **Verify**, the average rises from **0.5491** to **0.8169**. **Rubric** also helps, especially for deep research: **DR-Bench** rises from **0.5454** to **0.7075**, and **RealDR** from **0.4051** to **0.4779**. However, the gains are not universal. Rubric underperforms Vanilla on **SurGE-Content** and **MA**, and **Reference + Rubric** is not additive: its overall accuracy is **0.5784**, slightly below **Reference** alone at **0.5843**. This is one of the benchmark’s central empirical findings: adding more evaluation guidance does not reliably solve long-form judging [2606.01629].

The comparison between **Qwen3-32B with thinking** and **without thinking** further shows that extra internal reasoning is not consistently beneficial. The thinking version improves on **RealDR** from **0.3702** to **0.4580** and on **WP-Bench** from **0.6113** to **0.6379**, but overall average drops from **0.5745** to **0.5241**. The appendix reports that rank-correlation metrics sometimes tell a different story and that **DeepSeek-V4-Flash Vanilla** achieves the best average rank-correlation performance, but the paper recommends **accuracy as the primary metric** [2606.01629].

## 6. Length sensitivity, failure modes, and position bias

LongJudgeBench includes a direct analysis of output length sensitivity. Using only the **Vanilla** setting, the authors split each dataset into four equal-size length bins \(Q1\)–\(Q4\). The result is not a single monotonic “longer is worse” law. **RealDR**, **SurGE**, and **Verify** generally show decreasing performance from shorter to longer bins. **DR-Bench** is non-monotonic, with performance peaking around \(Q2\). **WP-Bench** shows no decreasing trend, and some models do better on longer responses. **MA** shows no clear monotonic pattern. The paper explicitly states that this supports its thesis that long-form evaluation is **not reducible to sequence length alone** [2606.01629].

The benchmark also documents concrete failure modes. In one computational chemistry deep research case, the actual user need concerned external electric-field direction when molecular orientation is uncertain. A candidate response discussed many related topics, including Gaussian field syntax and VASP, but only minimally addressed the core issue. **Qwen3-Max** scored it **9.26**, while humans scored **5.87**. In a technical survey case on Anthropic’s **Streamable HTTP**, a candidate mainly discussed general SSE-based streaming in the Messages API; **Qwen3-Max** treated these as equivalent and scored **8.85**, while humans scored **5.37**. These examples are used to argue that judges may overvalue surface richness, structure, and topic breadth, or may fail at fine-grained concept disambiguation in specialized domains [2606.01629].

Position bias remains severe on pairwise tasks. Under **Vanilla**, the appendix reports inconsistent underlying preferences after answer-order swaps at rates such as **78.7%** on **WP-Bench** and **48.9%** on **MA** for **GPT-4o-mini**, **43.0%** and **38.9%** for **Qwen3-Max**, **21.3%** and **37.8%** for **Qwen3-32B**, **16.0%** and **21.1%** for **Qwen3-32B w/o thinking**, and **10.6%** and **23.3%** for **Kimi-K2.6**. The benchmark also records practical breakdowns from **context-window overflow** and **safety-policy rejection**, especially in settings that combine long outputs with references and rubrics, and it counts these failures as incorrect in accuracy calculations [2606.01629].

A common misconception would be that references or rubrics fully stabilize long-form judging. The benchmark contradicts that view: rubrics and references are often helpful, but they are **not always sufficient**, and sometimes they reduce consistency rather than improving it.

## 7. Place in the judge-benchmark literature

LongJudgeBench sits within a broader shift from short-form judge evaluation to the study of complex, open-ended, and high-cost judgments. Earlier work such as **JudgeBench** focuses on challenging response pairs in knowledge, reasoning, math, and coding and shows that many strong judges perform near **random guessing (50%)** on correctness-based pairwise judgments, but it is not a long-form benchmark [2410.12784]. **JudgmentBench** is especially relevant methodologically: in a high-expertise legal domain, it compares rubric-based scoring with comparative judgment on the same tasks, same outputs, and same experts, finding that comparative judgments recover the intended quality ordering substantially better than rubrics, with mean Spearman’s rank correlation **0.908** versus **0.150**, and in less than half the annotation time [2605.25240]. This suggests that supervision protocol is itself a central design variable in complex-output evaluation.

The reliability issues documented in LongJudgeBench also align with parallel work on judge bias and aggregation. Studies of position bias define metrics such as **repetition stability**, **position consistency**, and **preference fairness**, and show that order effects in LLM judges are systematic rather than random [2406.07791]. Judge-aware ranking work extends Bradley–Terry-Luce with judge-specific discrimination parameters \(\gamma_k\), arguing that benchmarks without ground truth should model heterogeneity among judges rather than weighting all judges equally [2601.21817]. Psychometric critiques of judge benchmarks go further, arguing that benchmark rankings can become “high-confidence rankings that are in fact largely noise” when rubric adherence and factor validity are weak [2509.20293], while judge-datasheet work frames judges as measurement instruments with properties such as dark current, positional false preference, stable cross-sensitivity, and criterion-dependent tie behavior [2606.15610].

Within that landscape, LongJudgeBench’s distinctive contribution is to move meta-evaluation into **realistic long-form scenarios** with **expert human reference judgments**, **multiple protocols**, and **document-level assessment demands**. Its main conclusion is sobering rather than maximalist: current LLM judges can perform moderately well in some anchored settings, especially with references, but they remain unstable, scenario-sensitive, and only partially aligned with humans on long-form outputs [2606.01629].

Source: https://www.emergentmind.com/topics/longjudgebench