LongJudgeBench: Evaluating Long-Form LLM Judges

• LongJudgeBench is a benchmark designed to meta-evaluate LLM-as-judge systems specifically for long-form outputs, averaging over 9,000 tokens.
• It employs a unified framework covering pointwise, pairwise, and listwise protocols to assess document-level qualities like structure, depth, and consistency.
• Empirical results reveal that even the best configurations only achieve moderate accuracy (around 67%), highlighting challenges such as protocol sensitivity and position bias.

Searching arXiv for LongJudgeBench and closely related judge-benchmark papers. LongJudgeBench is a benchmark for meta-evaluating LLM-as-a-judge systems on long-form outputs across diverse scenarios and judgment protocols. It was introduced to address a gap in prior judge benchmarks, which the paper characterizes as being focused mainly on short-form outputs: benchmarks such as MT-Bench, LLMBar, RewardBench, PreferenceBench, RPR, JudgeBench, and RewardBench 2 mostly contain outputs only a few hundred tokens long, whereas LongJudgeBench’s candidate outputs average 9,249.7 tokens. The benchmark’s central claim is that long-form evaluation is not merely a matter of output length; it requires document-level assessment of overall organization, task-relevant coverage and depth, cross-section consistency, presentation quality, and scenario-specific quality criteria. LongJudgeBench therefore evaluates whether current LLM judges can reliably assess realistic long-form generations under expert human reference judgments rather than LLM-generated pseudo-gold labels (Chen et al., 1 Jun 2026).

1. Conceptual basis and formalization

LongJudgeBench treats each evaluation instance as

$x_i = (\mathcal{I}_i, \mathcal{C}_i, z_i^{*}),$

where $\mathcal{I}_i$ is the instruction, $\mathcal{C}_i$ is the content to evaluate, and $z_i^{*}$ is the reference judgment. The instruction is formalized as

$\mathcal{I} \doteq \{q, m\} \cup \mathcal{A},$

with $q$ the original generation query or task, $m$ the judging protocol, and $\mathcal{A}$ optional auxiliary information such as rubrics, weights, references, domain constraints, or formatting requirements. The content depends on the protocol: $$\mathcal{C} = \begin{cases} \{y_1\}, & \text{pointwise scoring},\ \{y_1, y_2\}, & \text{pairwise comparison},\ \{y_1, \ldots, y_K\}, & \text{listwise ranking}. \end{cases}$$ A judge $J_\theta$ produces

$\mathcal{I}_i$0

and the response may contain both explanation and final judgment,

$\mathcal{I}_i$1

This formulation makes the benchmark explicitly multi-protocol: it covers pointwise scoring, pairwise comparison, and listwise ranking within a single meta-evaluation framework (Chen et al., 1 Jun 2026).

The benchmark is motivated by the claim that long-form evaluation differs qualitatively from short-form evaluation. The issue is not only that outputs are longer, but that judges must assess holistic document properties such as structure, depth, consistency across sections, and domain-specific criteria. The paper also positions LongJudgeBench against concurrent work such as Long-form RewardBench, arguing that such resources are less suitable for meta-evaluating judges because many gold labels are produced through an LLM-judge pipeline rather than expert humans, and because the responses are shorter, around 3K tokens rather than LongJudgeBench’s 9K-plus average. This suggests that LongJudgeBench is designed as a benchmark for judge reliability under realistic long-form conditions, rather than merely as a longer version of short-answer judge evaluation (Chen et al., 1 Jun 2026).

2. Scope, scenarios, and dataset composition

LongJudgeBench spans five scenarios and six datasets, across Chinese and English, with 1,944 instances, 1,966 outputs, and an average output length of 9,249.7 tokens. The scenarios are deep research, scientific survey, creative writing, long-chain analysis, and systematic review (Chen et al., 1 Jun 2026).

The deep research scenario contains two datasets. DR-Bench is Chinese, with 200 instances / 200 outputs, uses pointwise evaluation, and has mean output length 18,012.9 tokens with median 9,604.5. RealDR (RealDeepResearch) is Chinese and English, with 640 instances / 640 outputs, also uses pointwise evaluation, and has mean output length 10,131.4 with median 9,610.0. This scenario targets long research reports with coverage, insight, structure, and realistic document features.

The scientific survey scenario uses SurGE, an English dataset with 82 instances / 164 outputs, evaluated listwise, with mean output length 28,758.0 and median 35,948.5. It tests whether judges can recover human rankings over scientific surveys for structure and content. The creative writing scenario uses a WritingPreferenceBench subset, denoted WP-Bench, with 526 instances / 526 outputs, evaluated pairwise, and mean output length 3,508.9 with median 2,178.5. The long-chain analysis scenario uses Verify, with 316 instances / 316 outputs, evaluated pointwise, and mean output length 3,052.6 with median 1,692.5. The systematic review scenario uses MA, with 180 instances / 120 outputs, evaluated pairwise, and mean output length 4,764.1 with median 4,515.5 (Chen et al., 1 Jun 2026).

These datasets cover domains including science and technology, finance, software, art, history, education, medicine, sociology, law, philosophy, environmental engineering, creative genres, and clinical medicine. A notable design choice is that LongJudgeBench preserves heterogeneous original protocols whenever possible and then normalizes them into a unified meta-evaluation setup. This suggests that the benchmark is intended to capture real long-form evaluation settings rather than a single synthetic task family (Chen et al., 1 Jun 2026).

3. Human reference judgments and scenario-specific supervision

A major feature of LongJudgeBench is its reliance on expert human judgments. In DR-Bench, the benchmark uses the open-source Chinese subset of DeepResearch Bench, consisting of 50 Chinese research tasks and 200 reports from four deep research agents: openai-deepresearch, gemini-2.5-pro-deepresearch, grok-deeper-search, and perplexity-Research. Human scoring is provided by 3 experts with relevant research backgrounds, who rate each report on 0–100 over four dimensions: comprehensiveness, insight, instruction following, and readability. The final score is a weighted sum of these dimensions, and the original benchmark reports 68.44% inter-annotator agreement (Chen et al., 1 Jun 2026).

RealDeepResearch is the main newly constructed dataset. It contains 40 tasks—29 Chinese and 11 English—across 12 disciplines, and each task is paired with 4 prompt styles and 4 generation models (Gemini, Grok, Mita, Perplexity), yielding 640 documents. Each document is scored by 2 annotators with relevant backgrounds on 0–10 across logical structure, presentation form, and bias checking; if disagreement is large, a third annotator adjudicates. The reported agreement is

$\mathcal{I}_i$2

which the authors interpret as high agreement. Total annotation cost was approximately USD 2,260 (Chen et al., 1 Jun 2026).

In SurGE, 41 topics in computer science each have 4 surveys—GT, AutoSurvey, ID, and Naive—ranked by 4 PhD students in computer science, supervised by 1 faculty member. Each survey is independently annotated by 2 experts on a 1–4 scale for structural quality and content quality, with agreement

$\mathcal{I}_i$3

In WP-Bench, the benchmark samples 263 human-validated preference pairs, doubled to 526 pairwise instances by swapping candidate order. Human annotation is performed by 11 professional writing evaluators who underwent an 8-hour rubric calibration process and score creativity, style, and emotional resonance on 0–3. Pairs require at least 2 of 3 annotators to agree on preference, ties are excluded, and the minimum score gap is

$\mathcal{I}_i$4

In VerifyBench-Hard, LongJudgeBench samples 316 instances with at least 2 annotators per instance and reported inter-annotator agreement 0.88–0.92. In MA, the benchmark uses 30 papers from clinical systematic review and meta-analysis data, compares three fixed system pairings, and obtains 180 pairwise evaluation instances after order swapping; judgments come from 8 annotators with medical backgrounds using a five-level preference scale A2, A1, tie, B1, B2 (Chen et al., 1 Jun 2026).

This supervision design matters because it anchors LongJudgeBench in expert human reference judgments across both objective and subjective scenarios. A plausible implication is that the benchmark is intended not only to compare judges, but also to expose where long-form evaluation remains dependent on domain expertise and protocol design.

4. Judge models, prompting settings, and metrics

The benchmark evaluates Qwen3-32B (without thinking), Qwen3-32B (with thinking), Qwen3-Max, GPT-4o-mini, GPT-5.2, DeepSeek-V4-Flash, GLM-5.1, and Kimi-K2.6. For each base model, four judging settings are tested: Vanilla, Rubric, Reference, and Reference + Rubric. All runs use the same task inputs, the same extraction procedure, and temperature = 0 for fairness and determinism (Chen et al., 1 Jun 2026).

The benchmark compares LLM judgments against expert human references under pointwise, pairwise, and listwise protocols. To make comparison consistent, it derives pairwise preferences where needed: directly from pairwise judgments, by comparing scalar scores for pointwise tasks, and by comparing ranks for listwise tasks. For pairwise tasks, candidate order is explicitly swapped and results are averaged to reduce position bias (Chen et al., 1 Jun 2026).

The primary metric is accuracy. For preference-based datasets, if $\mathcal{I}_i$5 is the set of non-tie human-labeled candidate pairs for query $\mathcal{I}_i$6, then

$\mathcal{I}_i$7

where tie predictions are counted as incorrect if the human label is non-tie. The benchmark also reports Spearman’s rank correlation and Kendall’s $\mathcal{I}_i$8. Given reference and predicted score vectors, Spearman correlation is computed from their induced ranks, and Kendall’s $\mathcal{I}_i$9 is

$\mathcal{C}_i$0

with $\mathcal{C}_i$1 and $\mathcal{C}_i$2 the numbers of concordant and discordant pairs, and $\mathcal{C}_i$3, $\mathcal{C}_i$4 ties occurring only in predicted or reference scores. The paper states that accuracy is the primary metric, while Spearman and Kendall are auxiliary (Chen et al., 1 Jun 2026).

5. Empirical results and scenario-specific behavior

LongJudgeBench’s headline result is that current LLM judges remain only moderately reliable on long-form output evaluation. Across all 32 model-setting combinations, only 12 exceed 0.60 average accuracy, and the overall mean is 0.5627. The best overall configuration is Qwen3-Max + Reference with average accuracy 0.6721, followed by DeepSeek-V4-Flash + Reference at 0.6626, and GLM-5.1 + Reference at 0.6485 (Chen et al., 1 Jun 2026).

Performance varies sharply by scenario. Verify is the easiest scenario, with average performance around 0.6742, and DeepSeek-V4-Flash + Reference reaches 0.8924. DR-Bench is also relatively strong, with average around 0.6454; rubric-based settings help substantially here, with DeepSeek-V4-Flash + Rubric and Kimi-K2.6 + Rubric both reaching 0.7200. By contrast, RealDR is much harder, with average around 0.4393 and best scores only in the high 0.5s; DeepSeek-V4-Flash + Rubric reaches 0.5710, and GLM-5.1 + Rubric reaches 0.5606. MA is also difficult, with average around 0.4983, while WP-Bench averages around 0.5776. SurGE is mixed, with some judges performing well on structure or content, but results remaining unstable across dimensions and settings (Chen et al., 1 Jun 2026).

The most consistent aid is Reference information. Overall average accuracy improves from 0.5313 in Vanilla to 0.5843 in Reference. On Verify, the average rises from 0.5491 to 0.8169. Rubric also helps, especially for deep research: DR-Bench rises from 0.5454 to 0.7075, and RealDR from 0.4051 to 0.4779. However, the gains are not universal. Rubric underperforms Vanilla on SurGE-Content and MA, and Reference + Rubric is not additive: its overall accuracy is 0.5784, slightly below Reference alone at 0.5843. This is one of the benchmark’s central empirical findings: adding more evaluation guidance does not reliably solve long-form judging (Chen et al., 1 Jun 2026).

The comparison between Qwen3-32B with thinking and without thinking further shows that extra internal reasoning is not consistently beneficial. The thinking version improves on RealDR from 0.3702 to 0.4580 and on WP-Bench from 0.6113 to 0.6379, but overall average drops from 0.5745 to 0.5241. The appendix reports that rank-correlation metrics sometimes tell a different story and that DeepSeek-V4-Flash Vanilla achieves the best average rank-correlation performance, but the paper recommends accuracy as the primary metric (Chen et al., 1 Jun 2026).

6. Length sensitivity, failure modes, and position bias

LongJudgeBench includes a direct analysis of output length sensitivity. Using only the Vanilla setting, the authors split each dataset into four equal-size length bins $\mathcal{C}_i$5–$\mathcal{C}_i$6. The result is not a single monotonic “longer is worse” law. RealDR, SurGE, and Verify generally show decreasing performance from shorter to longer bins. DR-Bench is non-monotonic, with performance peaking around $\mathcal{C}_i$7. WP-Bench shows no decreasing trend, and some models do better on longer responses. MA shows no clear monotonic pattern. The paper explicitly states that this supports its thesis that long-form evaluation is not reducible to sequence length alone (Chen et al., 1 Jun 2026).

The benchmark also documents concrete failure modes. In one computational chemistry deep research case, the actual user need concerned external electric-field direction when molecular orientation is uncertain. A candidate response discussed many related topics, including Gaussian field syntax and VASP, but only minimally addressed the core issue. Qwen3-Max scored it 9.26, while humans scored 5.87. In a technical survey case on Anthropic’s Streamable HTTP, a candidate mainly discussed general SSE-based streaming in the Messages API; Qwen3-Max treated these as equivalent and scored 8.85, while humans scored 5.37. These examples are used to argue that judges may overvalue surface richness, structure, and topic breadth, or may fail at fine-grained concept disambiguation in specialized domains (Chen et al., 1 Jun 2026).

Position bias remains severe on pairwise tasks. Under Vanilla, the appendix reports inconsistent underlying preferences after answer-order swaps at rates such as 78.7% on WP-Bench and 48.9% on MA for GPT-4o-mini, 43.0% and 38.9% for Qwen3-Max, 21.3% and 37.8% for Qwen3-32B, 16.0% and 21.1% for Qwen3-32B w/o thinking, and 10.6% and 23.3% for Kimi-K2.6. The benchmark also records practical breakdowns from context-window overflow and safety-policy rejection, especially in settings that combine long outputs with references and rubrics, and it counts these failures as incorrect in accuracy calculations (Chen et al., 1 Jun 2026).

A common misconception would be that references or rubrics fully stabilize long-form judging. The benchmark contradicts that view: rubrics and references are often helpful, but they are not always sufficient, and sometimes they reduce consistency rather than improving it.

7. Place in the judge-benchmark literature

LongJudgeBench sits within a broader shift from short-form judge evaluation to the study of complex, open-ended, and high-cost judgments. Earlier work such as JudgeBench focuses on challenging response pairs in knowledge, reasoning, math, and coding and shows that many strong judges perform near random guessing (50%) on correctness-based pairwise judgments, but it is not a long-form benchmark (Tan et al., 2024). JudgmentBench is especially relevant methodologically: in a high-expertise legal domain, it compares rubric-based scoring with comparative judgment on the same tasks, same outputs, and same experts, finding that comparative judgments recover the intended quality ordering substantially better than rubrics, with mean Spearman’s rank correlation 0.908 versus 0.150, and in less than half the annotation time (Yang et al., 24 May 2026). This suggests that supervision protocol is itself a central design variable in complex-output evaluation.

The reliability issues documented in LongJudgeBench also align with parallel work on judge bias and aggregation. Studies of position bias define metrics such as repetition stability, position consistency, and preference fairness, and show that order effects in LLM judges are systematic rather than random (Shi et al., 2024). Judge-aware ranking work extends Bradley–Terry-Luce with judge-specific discrimination parameters $\mathcal{C}_i$8, arguing that benchmarks without ground truth should model heterogeneity among judges rather than weighting all judges equally (Xu et al., 29 Jan 2026). Psychometric critiques of judge benchmarks go further, arguing that benchmark rankings can become “high-confidence rankings that are in fact largely noise” when rubric adherence and factor validity are weak (Feuer et al., 24 Sep 2025), while judge-datasheet work frames judges as measurement instruments with properties such as dark current, positional false preference, stable cross-sensitivity, and criterion-dependent tie behavior (Usami et al., 14 Jun 2026).

Within that landscape, LongJudgeBench’s distinctive contribution is to move meta-evaluation into realistic long-form scenarios with expert human reference judgments, multiple protocols, and document-level assessment demands. Its main conclusion is sobering rather than maximalist: current LLM judges can perform moderately well in some anchored settings, especially with references, but they remain unstable, scenario-sensitive, and only partially aligned with humans on long-form outputs (Chen et al., 1 Jun 2026).