Large Language Model-as-a-Judge (LaaJ)

Updated 13 July 2026

Large Language Model-as-a-Judge (LaaJ) is a framework that employs LLMs to assess generated outputs using pairwise, point-wise, rubric, and agent-based methods.
It enables scalable, explanation-rich judgments that replace or augment human annotations for benchmarking, preference learning, and safety evaluation.
Recent advances integrate SFT, DPO, and meta-evaluation metrics to enhance calibration, mitigate bias, and support effective policy optimization in various domains.

LLM-as-a-Judge (LaaJ) denotes the use of a LLM as an evaluator of generated outputs rather than as a generator of final task responses. In the literature, this evaluator may compare two candidate answers, assign a point-wise score, verify whether a rubric is satisfied, or inspect an action trace produced by an agentic system. The common objective is to replace or augment human annotation with scalable, explanation-producing judgments that can support benchmarking, preference learning, safety assessment, and policy optimization. Recent work increasingly treats judging not as an ad hoc prompting trick but as a trainable and measurable capability with its own failure modes, calibration requirements, and domain-specific constraints (Yu et al., 17 Feb 2025, Biskupski et al., 23 Mar 2026, Yu, 5 Aug 2025).

1. Core definitions and task formulations

The simplest formalization treats LaaJ as a function that consumes a task context and one or more candidate outputs, then emits a verdict together with textual reasoning. In pairwise settings, a generative judge compares two responses to the same query, produces a step-by-step analysis of strengths and weaknesses, and emits a binary preference such as “A is better” or “B is better.” In point-wise settings, the judge assigns a numerical score to a single answer. In rubric verification, the judge decides whether an output satisfies a human-written criterion. In agentic settings, the judge may evaluate not only the final output but also an action trace, tool usage, and per-step process quality (Yu et al., 17 Feb 2025, Gao et al., 14 Oct 2025, Biskupski et al., 23 Mar 2026, Yu, 5 Aug 2025, Peng et al., 29 Jun 2026).

Regime	Input	Output
Pairwise comparison	Query + two answers	Binary preference + analysis
Point-wise scoring	$(Q, A, R)$	$s \in [1,10]$
Rubric verification	Output + rubric	$\hat y_i \in \{0,1\}$
Agent-as-a-judge	$(x,\tau)$	$s \in \mathbb{R}^d$

These formulations induce different evaluation semantics. Pairwise judging emphasizes relative preference and is closely tied to preference learning. Point-wise judging emphasizes absolute score assignment under a rubric. Rubric verification decomposes broad judgments into binary checks over concrete criteria. Agent-as-a-judge extends the paradigm to long-horizon trajectories in which correctness may depend on intermediate tool calls, consistency across sections, or the handling of explicit constraints. This suggests that “judge quality” is not a single scalar property but a family of task-conditioned behaviors.

The literature also distinguishes the judge model from the evaluated or “victim” model. One formulation defines a dataset $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ , where $a_i = M_0(q_i)$ is the victim model’s answer and $y_i \in \{0,1\}$ is a human label, and writes the judge as

$\hat y_i, e_i = J_{\theta,P}(q_i, a_i),$

where $P$ is a judge prompt encoding the criteria and output format. In other work, the point-wise judge is written as

$s \in [1,10]$ 0

The agent-as-a-judge review generalizes this to

$s \in [1,10]$ 1

with multi-criterion outputs and optional tool access (Biskupski et al., 23 Mar 2026, Gao et al., 14 Oct 2025, Yu, 5 Aug 2025).

2. Judge construction, training, and calibration

A prominent construction strategy treats judge ability as a general ability of LLMs rather than a narrow evaluator head. "Improve LLM-as-a-Judge Ability as a General Ability" trains a judge in two stages: supervised fine-tuning (SFT) warm-up followed by direct preference optimization (DPO) enhancement. The SFT stage uses Math-PRM800K, Skywork-Reward-Preference-80K and subsets such as HelpSteer2, UltraFeedback, OffsetBias, and WildGuardMix, plus a small amount of general chat QA to prevent catastrophic forgetting. Training examples are pairs $s \in [1,10]$ 2, where $s \in [1,10]$ 3, and the loss is

$s \in [1,10]$ 4

The DPO stage samples multiple judgments on hard cases, forms chosen/rejected pairs $s \in [1,10]$ 5, and optimizes

$s \in [1,10]$ 6

with an auxiliary NLL term and

$s \in [1,10]$ 7

The same work uses GPT-4o to rewrite judge instructions along language, principle complexity, presentation format, and output format, swaps answer order to mitigate position bias, and balances answer lengths to mitigate length bias. Its 32B model, trained on 20 K SFT + 20 K DPO examples, reports an average RewardBench score of 92.7, with 96.6 on Chat, 83.3 on Chat-Hard, 91.9 on Safety, and 98.8 on Reasoning, while using only 2 %–40 % of the data employed by leading prior works (Yu et al., 17 Feb 2025).

A distinct line of work separates qualitative reasoning from quantitative calibration. "Quantitative LLM Judges" keeps a base judge frozen, embeds its textual critique $s \in [1,10]$ 8, and learns a lightweight post-hoc model

$s \in [1,10]$ 9

to predict human scores. The framework defines four variants: Least-Squares (LS), Multinomial (MN), Bradley–Terry–Luce (BTL), and Two-Headed BTL (BTL2). Reported training cost is 10–70× lower than full SFT, and on the Prometheus base judge LS reduces MSE from 6.346 to 2.626, while BTL2 reaches accuracy 0.783 and F1 0.721 on preference prediction, approaching or surpassing stronger raw judge baselines on several metrics (Sahoo et al., 3 Jun 2025).

Human-centered engineering systems have also emerged around LaaJ workflows. EvalAssist exposes direct-assessment and pairwise-comparison interfaces, serializes criteria in a JSON-like schema, and operationalizes a three-step prompt chain of assessment, answer selection, and reasoning summarization. Its pairwise mode aggregates all $\hat y_i \in \{0,1\}$ 0 comparisons through a win-rate

$\hat y_i \in \{0,1\}$ 1

while surfacing positional-bias warnings when judgments change under shuffling (Ashktorab et al., 2 Jul 2025).

3. Benchmarks, metrics, and measured performance

The recent literature has moved from reporting judge outputs on downstream tasks to explicit meta-evaluation against human labels. One large study evaluates 37 conversational LLMs with 5 judge-prompt templates over eight binary judgment tasks: Harmful Behavior, Illegal Activity, Personal Information, Brand Harm, Malicious Code/Text, Jailbreak Attacks, Similarity, and Truthfulness. After structured-output filtering and staged pruning, the best full-dataset result is GPT-4o with a Chain-of-Thought prompt at $\hat y_i \in \{0,1\}$ 2; open-source models with $\hat y_i \in \{0,1\}$ 3B parameters achieve $\hat y_i \in \{0,1\}$ 4; Qwen2.5 14B with CoT reaches $\hat y_i \in \{0,1\}$ 5. Five repeats at temperature 0.0 yield percent-agreement $\hat y_i \in \{0,1\}$ 6 across all 12 retained judge configurations, while second-level self-correction almost always worsens $\hat y_i \in \{0,1\}$ 7, with drops up to 0.28 for weaker models (Biskupski et al., 23 Mar 2026).

Rubric verification introduces a stricter meta-evaluation target: not whether the judge can produce a plausible score, but whether it can correctly verify a single rubric in long and complex outputs. RuVerBench contains 2,458 instances spanning Deep Research and Agentic Coding, with average output lengths of 7.1 K and 49.4 K tokens respectively and gold labels established through double independent annotation plus adjudication. The primary metric is Average Balanced Accuracy,

$\hat y_i \in \{0,1\}$ 8

On Deep Research, Gemini-3.1 Pro Preview reaches 94.7%; on Agentic Coding, GPT-5.4 reaches 89.4%. Yet the best score in Agentic Coding still leaves a 10.6-point gap to perfect verification, and batching multiple rubrics into one call can reduce accuracy by up to 20–30 points for $\hat y_i \in \{0,1\}$ 9 in the longer domain. Majority voting improves performance, especially in Agentic Coding, with gains of +3 to +4.5 points by 5 votes before plateauing (Peng et al., 29 Jun 2026).

Coding-specific benchmarking reaches similar conclusions about both promise and instability. CodeJudgeBench defines 4,260 pairwise instances over code generation, code repair, and unit test generation and finds that recent thinking models substantially outperform non-thinking models. Claude 4-Sonnet reports 80.2% average accuracy, Gemini 2.5-Pro 82.0%, and Qwen3-8B 65.6%, while some specially trained non-thinking judges remain near the mid-50s. At the same time, pairwise accuracy can shift by up to 11% under answer-order reversal, point-wise prompting produces tie rates around 45–60 % on CodeGen, and retaining comments and reasoning in the full response improves judge performance relative to stripped code-only inputs (Jiang et al., 14 Jul 2025).

For process-oriented code evaluation, another approach generates a benchmark automatically from a graph of artifact types and self-consistency cycles. With 232 valid “true” pairs and an equal number of random “false” pairs, two candidate judges each reached 100 % accuracy on same/different judgments and were fully symmetric. On more difficult clusters, final scores were 99.54 % and 99.85 % respectively. This benchmark design treats cycles in the graph as automatically generated correctness claims, allowing the judge itself to be validated against formal self-consistency properties rather than only against human preference labels (Farchi et al., 2024).

4. Bias, calibration, and statistical reliability

Bias analysis has become central to LaaJ research. In communication systems under point-wise scoring, one study defines a bias coefficient

$(x,\tau)$ 0

and examines 11 biases spanning implicit style-based effects and explicit content-based effects. The implicit set includes Rich Content (Elaboration), Verbosity (Length), Chain-of-Thought, and Sentiment (Tone) bias. The explicit set includes Authority, Factual Error, Diversity/Gender, Bandwagon, Distraction, and Compassion-Fade (Source Identity) bias. Under a detailed rubric, GPT-Judge and JudgeLM assign lower scores to verbose or CoT-style answers, with $(x,\tau)$ 1, $(x,\tau)$ 2, and $(x,\tau)$ 3. Factual errors receive the strongest penalty, with $(x,\tau)$ 4 versus $(x,\tau)$ 5, and strict rubric prompts also penalize authority bias with $(x,\tau)$ 6. The same work reports that a detailed scoring rubric reverses some superficial-feature preferences seen under minimal prompts, and proposes robust prompt design, automated bias detection, calibration, specialized fine-tuning, and ensemble judging as mitigation strategies (Gao et al., 14 Oct 2025).

A related mitigation line treats “superficial quality” as a quantity that can be subtracted from the raw judge signal. For closed-source judges, the calibrated score is

$(x,\tau)$ 7

where $(x,\tau)$ 8 estimates fluency or stylistic quality. For open-source judges, contrastive training introduces fluent but off-instruction negatives. On the LLMBar benchmark, text-davinci-003 with probability-level calibration raises adversarial average accuracy from 48.9 to 67.0 or from 43.0 to 66.9 depending on access mode, GPT-4 with prompt calibration increases adversarial average from 65.2 to 69.8, and Vicuna-7B with contrastive answer-level training improves adversarial average from 39.7 % to 57.2 % (Zhou et al., 2024).

Scoring-based LaaJ exhibits additional prompt-template sensitivity. "Evaluating Scoring Bias in LLM-as-a-Judge" defines scoring bias as a shift under perturbations to rubric order, score IDs, or reference answers. Across GPT-4o, DeepSeek-V3-671B, Qwen3-32B, Qwen3-8B, and Mistral-24B, even GPT-4o shows up to $(x,\tau)$ 9 under simple rubric-order or ID changes and up to $s \in \mathbb{R}^d$ 0 when a non-full reference is provided. Smaller judges can suffer $s \in \mathbb{R}^d$ 1 under rubric or ID bias and up to $s \in \mathbb{R}^d$ 2 under poorly chosen reference-answer scores. The paper’s practical recommendation is to test ascending and descending rubric order, empirically compare score ID styles, and include a reference answer flagged as full mark while avoiding low-score references that drag the scoring distribution downward (Li et al., 27 Jun 2025).

Beyond prompt-level bias, the estimation problem itself can be biased. "Bias and Uncertainty in LLM-as-a-Judge Estimation" models judge outputs as noisy measurements of true labels with sensitivity $s \in \mathbb{R}^d$ 3 and specificity $s \in \mathbb{R}^d$ 4, yielding

$s \in \mathbb{R}^d$ 5

It defines judge quality as Youden’s index

$s \in \mathbb{R}^d$ 6

and cross-model calibration instability as

$s \in \mathbb{R}^d$ 7

The paper shows that naive estimates are systematically biased, shared-calibration bias correction can become unstable or even wrong-signed when $s \in \mathbb{R}^d$ 8 is low or $s \in \mathbb{R}^d$ 9, and model-comparison estimates can reverse sign with high apparent confidence. Its reporting guidance is to publish the estimand, estimator, calibration design, confidence intervals, and diagnostics for $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 0, $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 1, $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 2, and $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 3; if $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 4 is low, for example below $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 5, corrected estimates become suspect (Fiedler, 7 May 2026).

5. Multilingual, coding, agentic, and domain-specific settings

Multilingual LaaJ remains notably unstable. A 25-language study over XQuAD, MGSM, WMT23, WikiLingua, and XDailyDialog reports mean Fleiss’ $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 6 across tasks and models, with GPT-4o on WikiLingua (Yes/No) reaching $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 7 and Aya-Expanse on WMT23 (Grade) falling to $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 8. Binary judgments are more consistent than 5-point grades, low-resource languages show especially weak agreement with English, and adding explanations increases $D = \{(q_i, a_i, y_i)\}_{i=1}^N$ 9 by approximately +0.05–0.10. A simple majority-vote ensemble over three open-source judges improves consistency in 7/10 settings (Fu et al., 18 May 2025).

Pairwise multilingual judging exhibits a more specific language bias. In same-language judging on MMMLU, European languages consistently outperform African languages, the bias is more pronounced in culturally related subjects, and answer language matters more than question language. In inter-language judging, most models favor English answers, and this preference is influenced more by answer language than by question language. The study tests whether this is reducible to low-perplexity bias and finds only weak Pearson correlations of roughly $a_i = M_0(q_i)$ 0 to $a_i = M_0(q_i)$ 1, with nested-model tests showing that language identity contributes significantly beyond perplexity alone (Zhou et al., 20 Jan 2026).

Construction choices matter in multilingual judge training as well. In English, Spanish, and Basque, partial translation that keeps rubrics and system prompts in English (“io_lang”) consistently outperforms full translation in in-domain evaluation. Multilingual supervision improves Pearson correlation in Spanish and Basque for 8B models, while monolingual fine-tuning remains best for English. The best open-weight in-domain result reported is Latxa-Inst-8B_ft at 0.836/0.724 on RECON en, nearly matching GPT-5.2 at 0.844/0.650, whereas in out-of-domain FLASK, zero-shot 70B models outperform fine-tuned alternatives and 70B fine-tuning can degrade performance (Zubiaga et al., 27 May 2026).

Agentic scenarios intensify long-context verification problems. RuVerBench shows that weaker models are more sensitive to prompt-style changes, especially in Agentic Coding, where Qwen3.5-27B gains +11.8 points Avg BAcc under Strict versus Default prompting. The same benchmark indicates low error overlap among top verifiers—16.1% in Deep Research and 20.6% in Agentic Coding—suggesting heterogeneous failure modes even at the frontier (Peng et al., 29 Jun 2026).

Domain-specific deployment adds governance constraints. In healthcare, a scoping review of 49 studies finds that evaluation and benchmarking dominate the field (37 studies, 75.5%), pointwise scoring is used in 42 studies (85.7%), GPT-family judges appear in 36 studies (73.5%), and risk-of-bias testing is absent in 36 studies (73.5%). Only 1 study (2.0%) examines demographic fairness, and none assess temporal stability or patient context. The proposed MedJUDGE framework organizes evaluation around validity, safety, and accountability across three clinical risk tiers and recommends mandatory alignment verification, broader safety validation for Tier B/C use cases, multi-family judge panels, and explicit error-propagation analysis in multi-judge pipelines (Li et al., 3 Apr 2026).

A legal application illustrates a different pattern: Arabic legal judgment prediction uses an LLM judge for an 8-dimension qualitative assessment comprising Accuracy, Relevance, Coherence, Brevity, Legal Language, Faithfulness, Clarity, and Consistency. In that setting, fine-tuned smaller models approach larger-model performance, but the authors explicitly note that the evaluation relied on an LLM judge that may exhibit its own biases and that human annotation validation is needed (Kmainasi et al., 15 Jan 2025).

6. Applications, extensions, and open research directions

LaaJ is no longer confined to offline benchmarking. In preference-learning pipelines, accurate judge models serve as the critic signal in RLHF or RLAIF, and judge-generated labels can materially alter downstream policy optimization. The two-stage RISE-Judge work reports that its judge signals improve an internal policy model from 7.20 to 7.60 average AlignBench, compared with 7.40 when GPT-4o is used as the judge, while also slightly improving general capability benchmarks from 83.1 to 83.4 average relative to Qwen2.5-32B-Instruct (Yu et al., 17 Feb 2025).

Safety and bias auditing provide another major application. CLEAR-Bias uses a judge to classify model responses into Stereotyped, Counter-stereotyped, Debiased, or Refusal and then computes category-specific robustness $a_i = M_0(q_i)$ 2, fairness $a_i = M_0(q_i)$ 3, and safety

$a_i = M_0(q_i)$ 4

The released dataset contains 200 base prompts plus 4,200 adversarial prompts over seven isolated and three intersectional bias dimensions. The benchmark reports strong variation across model families and attack types, with machine translation, refusal suppression, and prompt injection producing the largest weighted average safety reductions (Cantini et al., 10 Apr 2025).

The paradigm is also expanding from single judges to multi-agent and meta-judging systems. A review of agent-as-a-judge traces the progression from automatic metrics such as BLEU and ROUGE, to single LLM judges, to debate and committee architectures including ChatEval, DEBATE, CourtEval, and MAJ-EVAL, and finally to tool-using process judges that can inspect action traces. On DevAI code tasks, an agent judge reportedly achieved 99.7 % agreement with human-majority, compared with 69 % for a single LLM judge. The same review summarizes typical trade-offs: single judges are cheapest; multi-agent systems cost roughly 2–4× more; tool-using agent judges cost roughly 10–20× more but provide per-step failure diagnosis (Yu, 5 Aug 2025).

Meta-judging generalizes this one level further by treating judgments themselves as objects of evaluation. The survey "Meta-Judging with LLMs" frames a meta-judge as an aggregator over primary judges’ scores and rationales, with mechanisms such as weighted averaging, majority voting, chain-of-thought aggregation, and self-consistency sampling. The survey attributes the rise of this paradigm to prompt sensitivity, systematic biases, verbosity effects, and unreliable rationales in single-judge systems. This suggests that future judge architectures may increasingly resemble ensembles or hierarchical evaluators rather than single prompted models (Silva et al., 24 Jan 2026).

A plausible implication of the recent literature is that LaaJ is evolving into an independent systems layer within LLM development. It now has its own training recipes, benchmarks, calibration theory, bias taxonomies, uncertainty diagnostics, multilingual transfer questions, human-centered tooling, and governance frameworks. The strongest current results show that high-performing judges can be highly data-efficient, correlate strongly with human assessments, and provide useful downstream alignment signals. At the same time, the recurring findings on position bias, language bias, scoring-template sensitivity, calibration instability, long-context rubric noise, and domain-governance gaps indicate that judge outputs should be treated as measurements requiring validation rather than as ground truth by default (Yu et al., 17 Feb 2025, Biskupski et al., 23 Mar 2026, Peng et al., 29 Jun 2026, Fiedler, 7 May 2026).