LLM-as-Judge Protocol

• LLM-as-Judge Evaluation Protocol is a standardized framework that measures model outputs with structured rubrics and aggregation methods.
• It emphasizes methodological rigor by incorporating repeated sampling, position-controlled prompting, and comprehensive bias checks.
• The protocol applies to diverse settings, from reference-free dialogue evaluation to long-form creative tasks, ensuring reliable and auditable judgments.

Searching arXiv for the cited survey and closely related papers on LLM-as-Judge reliability, bias, and protocol design. LLM-as-Judge evaluation protocol denotes a standardized procedure for using a LLM as an evaluator of model outputs, agent behavior, or other artifacts under explicit rubrics, prompting constraints, replication rules, aggregation methods, and reporting requirements. In the contemporary literature, the protocol is defined not merely by a scoring prompt but by a full measurement pipeline whose central objectives are consistency, validity, fairness, and scalability. The protocol therefore spans task taxonomy, rubric design, position-controlled prompting, repeated sampling, bias and robustness checks, calibration, and reproducibility artifacts, with the overarching aim of making judge outputs attributable, auditable, and aligned with expert or human judgments across text and multimodal settings (Gu et al., 2024, Yang et al., 9 Jul 2026).

1. Conceptual basis and objectives

The modern protocol treats an LLM judge as a measurement instrument rather than as a one-off heuristic scorer. The survey literature identifies four primary objectives. Consistency requires stable judgments across prompts, seeds, and rounds, supported by structured outputs and aggregation. Validity requires alignment with expert or human gold judgments across domains and tasks. Fairness requires minimizing documented biases, including position or order bias, verbosity or length bias, concreteness or citation bias, self-enhancement or model identity bias, sentiment bias, and diversity-related bias. Scalability requires low-cost, reproducible evaluation with robust open-source alternatives and batching (Gu et al., 2024).

Later reliability work sharpens this framing. “Evaluator-replacement ambiguity” denotes the fact that, even when candidate responses are fixed, replacing the judge can move the reported scores. This is treated as a measurement-validity problem rather than a simple model-upgrade question. In that framing, reliability is multi-dimensional: it includes judgment validity, bias robustness, aggregation independence, and protocol auditability. A score change after switching judges is therefore ambiguous unless the pipeline is instrumented well enough to attribute the change (Yang et al., 9 Jul 2026).

A parallel critique targets the common practice of equating agreement with objectivity. “Evaluation Illusion” describes a failure mode in which judges generate sophisticated critiques but anchor scores on shared surface heuristics such as formatting, fluency, confident tone, length, or structure instead of substantive quality. In a large-scale study of 105,600 evaluation instances, near-perfect model-level agreement, with Spearman $\rho \approx 0.99$, coexisted with substantially weaker sample-level agreement, with mean Pearson $\bar{r} \approx 0.72$ and absolute agreement $\mathrm{ICC} \approx 0.67$. This gap implies that macro-level agreement can mask micro-level fragility (Song et al., 11 Mar 2026).

A further methodological correction concerns metric choice. Exact-match agreement systematically overstates discriminative ability because it does not correct for chance; the reported “kappa deflation” between exact match and Cohen’s $\kappa$ is universal in a large cross-provider study, reaching 33.8–41.3 percentage points on MT-Bench. This supports the view that LLM-as-Judge protocol design must center chance-corrected agreement, not only raw agreement, and must separate reliability from validity rather than conflate them (Norman et al., 17 Jun 2026).

2. Evaluation settings and task taxonomy

The protocol literature organizes LLM-as-Judge settings along several orthogonal axes: whether evaluation is reference-free or reference-based, whether decisions are expressed as absolute scoring or pairwise ranking, whether the rubric is single-criterion or multi-criterion, and which domain is under evaluation (Gu et al., 2024).

Reference-free evaluation judges quality without gold references and is used for dialogue helpfulness, safety or harms, and open-ended writing. Reference-based evaluation compares outputs to a source, specification, or reference and is used for summarization, image captioning, factuality, and coding tasks. In reference-based settings, factuality and adherence can often be operationalized more directly, including through atomic checks in the style of FActScore, caption-oriented protocols such as CLAIR or FLEUR, or specification-driven coding evaluation such as DevBench (Gu et al., 2024).

Absolute scoring and pairwise ranking behave differently. Scoring, using Likert or numeric scales, is suitable for rubric-based multi-criterion evaluation and dashboards, but is more sensitive to biases. Pairwise evaluation, where the judge chooses the better of two responses with a tie option, is more aligned with human judgments and more robust to bias; it is therefore recommended when feasible (Gu et al., 2024). In multilingual evaluation, the literature has focused mainly on pointwise judging because assembling parallel multilingual candidate pairs is difficult, but this comes with weak cross-language consistency, with average Fleiss’ $\kappa$ approximately 0.3 across tasks and languages (Fu et al., 18 May 2025).

Long-form evaluation adds another layer of task specialization. It requires document-level assessment of organization, coverage and depth, cross-section consistency, and scenario-specific quality criteria rather than merely sentence-level correctness. LongJudgeBench formalizes this across deep research, scientific survey, creative writing, long-chain analysis, and systematic review, with outputs averaging 9,249.7 tokens, showing that long-form judging is not reducible to short-form protocols (Chen et al., 1 Jun 2026).

Axis	Main options	Typical use
Reference relation	Reference-free; reference-based	Dialogue, safety, open-ended writing; summarization, coding, factuality
Decision format	Absolute scoring; pairwise ranking	Dashboards and criterion scores; human-aligned comparison
Rubric structure	Single-criterion; multi-criterion	Helpfulness only; accuracy, coherence, factuality, safety
Data modality	Text; multimodal	Writing, coding, dialogue; captioning, image+text evaluation

Domain coverage in the survey literature includes coding and software, writing quality and summarization, factuality and verifiability, safety and toxicity, dialogue helpfulness, and multimodal caption quality and consistency. Real deployments also extend to multilingual evaluation, long-form evaluation, human–AI co-creation in software engineering, and web development, each of which introduces domain-specific reliability hazards (Gu et al., 2024, Amin et al., 30 Apr 2026, Li et al., 21 Oct 2025).

3. End-to-end protocol design

An end-to-end protocol begins with task and dataset selection. The task is first mapped to the evaluation taxonomy, and pairwise evaluation is preferred where feasible. Dataset construction is then driven by coverage and representativeness: MTBench or Chatbot Arena samples for dialogue, diverse topical sets for writing or summarization, FActScore-style atomic statements for factuality, DevBench tasks with specifications and unit tests for coding, and CLAIR or FLEUR-style tasks for multimodal caption evaluation. The survey protocol further recommends balancing difficulty and domain coverage and including adversarial variants, such as learned phrase attacks, sentiment or style perturbations, and bias-controlled variants such as length and citation manipulations (Gu et al., 2024).

Rubric design and standardization follow. Criteria must be defined with anchors, exemplars, and counter-exemplars; scales may be 1–5, 1–10, or 0–100 for absolute scoring, and 2-option or 3/4-option with tie or both-good/both-bad for pairwise evaluation. For multi-criterion scoring, the survey specifies a weighted score

$S = \sum_{k=1}^{K} w_k s_k,$

where $s_k$ are criterion scores and $w_k \ge 0$ with $\sum_k w_k = 1$ (Gu et al., 2024). Hierarchical decomposition is recommended for ambiguous dimensions.

The judge prompt is then standardized. A canonical system instruction is: “You are an impartial evaluator. Follow the rubric exactly. Output strictly in JSON.” Reference-free prompts include the task, the response or responses, and the rubric while hiding model identity; reference-based prompts additionally include the source or reference and specify the allowed use of that reference. Structured templates cover absolute scoring, pairwise decisions, and binary yes/no checks, and few-shot exemplars are used to demonstrate the expected JSON format and tie usage. Temperature is typically fixed at zero for determinism in judging, and self-consistency is obtained by running $N \ge 3$ passes with paraphrased rubrics or shuffled positions (Gu et al., 2024).

Leakage and contamination controls prohibit self-identification and model-name references in the prompt, track provenance through model versions and prompt hashes, keep training and evaluation sets disjoint when a judge is fine-tuned, and explicitly log dataset splits. The same logic motivates strict parser and validator versioning, fallback logging, and round-by-round traces in debate-based systems; without such logs, shifts induced by judge debate or protocol changes cannot be causally attributed to reasoning rather than parsing artifacts (Yang et al., 9 Jul 2026).

Sampling and replication are protocol-level requirements rather than optional extras. For pairwise evaluation, at least two position orders, A/B and B/A, are run per item; if explanations are used, protocol variants may additionally use multiple rubrics. For scoring, independent repeated runs are required, and repeated passes remain useful even at temperature zero because the goal is stability assessment, not only stochastic averaging (Gu et al., 2024).

Aggregation is task-dependent. Pairwise judgments can be aggregated by majority vote across passes and swapped positions, with ties escalated to a stronger judge or to a human. Scores can be aggregated by mean clipping to rubric bounds. For system ranking, the survey also lists Bradley–Terry, Thurstone Case V, and Elo-style updates as principled aggregation mechanisms (Gu et al., 2024).

Human-centered work introduces an additional procedural layer: criteria iteration on representative subsets, blind human review, agreement inspection, and reconfiguration before scaling. This is especially important when references are absent or stakeholder intent is evolving, as in headline generation or other open-ended tasks (Pan et al., 2024).

4. Reliability, validity, and calibration

Reliability assessment in LLM-as-Judge protocols spans agreement, consistency, ranking concordance, calibration, and significance testing. The survey recommends percentage agreement,

$\bar{r} \approx 0.72$0

Cohen’s $\bar{r} \approx 0.72$1,

$\bar{r} \approx 0.72$2

Krippendorff’s $\bar{r} \approx 0.72$3,

$\bar{r} \approx 0.72$4

intraclass correlation using $\bar{r} \approx 0.72$5 for continuous scores, Spearman’s $\bar{r} \approx 0.72$6 for system-level rankings, and Kendall’s $\bar{r} \approx 0.72$7 for pairwise ranking concordance (Gu et al., 2024).

Position-specific reliability is measured separately. The survey defines

$\bar{r} \approx 0.72$8

with conflict rate equal to $\bar{r} \approx 0.72$9 (Gu et al., 2024). This matters because high test–retest reliability can coexist with large position bias, a pattern described as a consistency–bias paradox in large-scale benchmark results (Norman et al., 17 Jun 2026).

Validity is evaluated against human or expert labels at both item and system levels. For categorical outputs, item-level accuracy or F1 is used; for scored outputs, RMSE or MAE; and for model rankings, Spearman’s $\mathrm{ICC} \approx 0.67$0. Significance testing can use permutation tests on agreement or $\mathrm{ICC} \approx 0.67$1 differences, $\mathrm{ICC} \approx 0.67$2-tests on mean score differences when assumptions hold, or Wilcoxon tests otherwise. Power analysis is used to estimate the sample size needed to reach a target confidence-interval width or minimum detectable effect (Gu et al., 2024).

Calibration is a distinct requirement because the survey identifies overconfidence and weak self-verification as recurring failure modes. When a judge reports confidence $\mathrm{ICC} \approx 0.67$3, calibration can be assessed with the Brier score,

$\mathrm{ICC} \approx 0.67$4

and Expected Calibration Error,

$\mathrm{ICC} \approx 0.67$5

Reliability diagrams are recommended, and conservative confidence reporting is preferred where overconfidence is documented (Gu et al., 2024).

Recent work also argues for judge-specific “datasheets.” The psychometric “dark current” protocol measures false preference under true-vacuum inputs, stable cross-sensitivity to same-quality surface variation, positional false preference, target sensitivity on a controlled quality ladder, and criterion shifts induced by tie instructions. This perspective emphasizes that a judge’s operating point can change without any increase in underlying resolution (Usami et al., 14 Jun 2026).

Quantitative post-hoc calibration is another response to the validity problem. “Quantitative LLM judges” treat the base judge’s textual evaluation and raw score or probability as features in regression or Bradley–Terry–Luce-style models, improving alignment with human scores while keeping the base judge frozen. This post-hoc modeling has been shown to improve the predictive power of existing judges with lower computational cost than supervised fine-tuning in the studied settings (Sahoo et al., 3 Jun 2025).

5. Bias, robustness, and security

Bias auditing is a core part of the protocol rather than an auxiliary evaluation. The survey identifies position bias, verbosity bias, concreteness or citation bias, self-enhancement or model identity bias, sentiment or style bias, and diversity or fairness concerns as recurring failure modes. The corresponding bias batteries therefore include order swapping, length padding without new content, citation insertion, identity masking versus revelation, tone or emoji variation, and demographic variants where relevant (Gu et al., 2024).

Adversarial robustness is closely related but not reducible to bias. The survey explicitly warns that trivial phrases can inflate scores and that “null model” traps can win pairwise comparisons. Robustness checks therefore include universal attack phrases, majority-opinion insertions, meaningless system-prompt statements, and null-model adversarial probes, alongside defensive baselines such as perplexity screens, format enforcement, position-swapped aggregation, and adjudication on conflicts (Gu et al., 2024).

The auditing literature shows that stronger judges reduce but do not remove bias. On LLMBar, for example, Qwen3-1.7B had a position-flip rate of 0.320, whereas MiniMax releases reduced flips to 0.117–0.147, but bias persisted; MiniMax-M2.7 still flipped 14.7% of cases. Verbosity bias under a fixed padding probe likewise fell from 0.547 for Qwen3-1.7B to roughly 0.13 for MiniMax releases, but was not eliminated. This supports the recommendation that even strong judges require explicit bias probes and A/B randomization (Yang et al., 9 Jul 2026).

Security work systematizes these concerns into a broader attack surface. LLM judges can be targets of prompt injection, token or formatting perturbations, training-time poisoning, rubric manipulation, and pipeline abuse through long irrelevant suffixes; they can also be used as instruments in downstream attacks if allowed to optimize rather than merely evaluate. Recommended safeguards include instruction hardening, rubric review committees, context sanitization, retokenization, judge isolation, deterministic decoding, adversarial stress tests, multi-judge committees, provenance tracking, and ecosystem-level monitoring (Masoud et al., 31 Mar 2026).

RobustJudge operationalizes this security perspective with an attack factory, defense guard, prompt-template optimization, and real-world deployment tests. It finds that LLM-as-Judge systems remain vulnerable to Combined Attack and PAIR, that retokenization and LLM-based detectors provide improved protection, that robustness is highly sensitive to prompt-template and judge-model selection, and that applying the framework to Alibaba’s PAI platform reveals previously unreported vulnerabilities (Li et al., 11 Jun 2025).

Bias also becomes more severe in multilingual and long-form settings. Multilingual LLM-as-Judge systems show weak cross-language consistency, particularly in low-resource languages, and neither multilingual training nor increased scale directly guarantees more reliable judgments (Fu et al., 18 May 2025). In long-form evaluation, rubrics and references help but are not universally sufficient, and scenario-specific instability remains substantial across deep research, surveys, creative writing, long-chain analysis, and systematic review (Chen et al., 1 Jun 2026).

6. Specialized protocols, applications, and open challenges

As LLM-as-Judge has expanded, increasingly specialized protocols have emerged. One response to generic-rubric fragility is knowledge-grounded rubric generation. MERG, or Metacognitive Enhanced Rubric Generation, activates domain knowledge, performs metacognitive bias reflection, generates task-specific rubrics with five dimensions and anchors at 2/5/8/10, and then scores with evidence citations and a bias-verification note. In codified domains, such as Education and Academic writing, agreement increases under this knowledge grounding, while in subjective domains agreement decreases, which is interpreted as surfacing genuine pluralism rather than reducing quality (Song et al., 11 Mar 2026).

A related line eliminates human-authored rubrics entirely by generating dataset-specific or instance-specific rubrics automatically. Training-free dynamic rubric generation uses the judge itself as rubric generator, while a fine-tuned rubric generator can be iteratively improved through meta-judge preference signals and Bradley–Terry aggregation over rubric candidates. In the reported experiments, instance-specific rubrics consistently outperform dataset-specific rubrics, and a fine-tuned 14B rubric generator outperforms a larger proprietary model at rubric generation (Wang et al., 28 May 2026).

Another line of work turns the judge into a trainable policy. FairJudge models judging behavior as a learnable, regularized policy optimized through a staged SFT–DPO–GRPO curriculum to improve adaptivity, debiasing, and cross-mode consistency. It constructs a dataset with explicit supervision for rubric adherence, non-semantic perturbation robustness, and pairwise–pointwise consistency, and reports higher agreement, F1, and consistency than several larger baselines on its benchmark suite (Yang et al., 6 Feb 2026).

Reference-based evaluation admits additional specialization. In settings with objectively verifiable answers, BERT-as-a-Judge replaces generative judges with a cross-encoder classifier initialized from EuroBERT 210M and trained on synthetically labeled question–candidate–reference triplets. It is designed for semantic correctness under phrasing and formatting variation, and in the reported experiments it outperforms lexical baselines while matching much larger LLM judges at much lower inference cost (Gisserot-Boukhlef et al., 10 Apr 2026).

Applications also impose domain-specific protocol changes. In human–AI co-creation for coding and software engineering, the protocol uses grouped train/validation/test splits by user and problem to avoid trajectory leakage, schema-constrained outputs with probability and rubric sub-scores, validation and repair logic, inter-judge agreement analysis, and trajectory-level measures such as Success-at-Turn, time-to-success, revision churn, and CodeBLEU. This extends LLM-as-Judge beyond static outputs to multi-turn collaborative processes (Amin et al., 30 Apr 2026).

Multi-agent judge architectures broaden the notion of evaluation itself. MAJ-EVAL automatically constructs evaluator personas from domain-relevant documents, instantiates them as agents with different evaluative dimensions, conducts in-group debate, and aggregates multi-dimensional feedback. In educational and medical domains this produces better alignment with human experts than several single-judge baselines, particularly for stakeholder-sensitive criteria (Chen et al., 28 Jul 2025).

Finally, test-time adaptation has become part of the protocol discussion. Learning While Evaluating (LWE) maintains an evolving meta-prompt that generates sample-specific evaluation instructions and refines itself through self-generated feedback, while Selective LWE updates only on self-inconsistent cases under response-order swapping. This suggests that some judge properties can improve during sequential deployment without external training data, although the approach remains tied to pairwise settings in the reported experiments (Jwa et al., 7 Dec 2025).

Across these variants, the open challenges remain stable. The literature repeatedly identifies susceptibility to phrasing and order, hallucinated rationales, residual position or length or concreteness bias, domain-knowledge gaps, weak calibration, universal adversarial phrases, and the need for stronger human-in-the-loop oversight. The most consistent conclusion is that LLM-as-Judge protocols can be reliable only when they are treated as full measurement systems—standardized, replicated, bias-audited, and transparently reported—rather than as single prompts wrapped around strong base models (Gu et al., 2024, Yang et al., 9 Jul 2026, Usami et al., 14 Jun 2026).