LLM as Evaluator: Methods and Challenges

• LLM as Evaluator is a paradigm where large language models assess outputs using predefined rubrics, role definitions, and contextual prompts to simulate human judgment.
• It leverages methods like direct scoring, pairwise comparison, and yes/no verification while exposing challenges in consistency, non-transitivity, and multilingual instability.
• Dynamic aggregation, refined prompt design, and transparent criteria are key strategies to improve evaluator reliability and mitigate biases across varied domains.

LLM as an evaluator, often termed LLM-as-a-Judge or LLM-as-an-Evaluator, denotes the use of a LLM to assess an object of interest—typically a model output, a student answer, a summary, a dialogue, or an explanation—and return an evaluation such as a score, label, rank, pairwise preference, or rationale. The survey literature formalizes this as a mapping from an input $x$ and context $\mathcal{C}$ to an evaluation outcome $\mathcal{E}$, written as $$\mathcal{E} \leftarrow \mathcal{P}_{\mathcal{LLM}(x \oplus \mathcal{C})}$$ (Gu et al., 2024). The paradigm has become prominent because reference-based metrics such as BLEU and ROUGE are often inadequate for open-ended generation, while human evaluation is expensive, slow, and difficult to scale (Gu et al., 2024). At the same time, recent work has shown that evaluator quality is constrained by inconsistency, non-transitivity, multilingual instability, prompt sensitivity, and model-specific bias, so the topic is now understood as a measurement problem rather than a simple prompting problem (Yu et al., 23 May 2025).

1. Definition and operational forms

In current usage, an evaluator LLM is a model prompted to judge output quality in a way that is intended to align with human instructions or expert criteria. In multilingual evaluation, this is described as a pipeline in which a context or prompt template $C$ is combined with the item $x$, passed to an LLM, and converted into a final evaluation result $p$; the prompt typically contains a role definition, an evaluation rubric, and an output format (Fu et al., 18 May 2025). The survey literature treats the resulting evaluation as a general-purpose surrogate for human judgment, capable of producing labels, scalar scores, rankings, or explanations (Gu et al., 2024).

Several operational modes recur across the literature. Direct assessment asks the judge to score or classify a single output against an explicit rubric. Pairwise comparison asks the judge to decide which of two outputs is better, often aggregating the results into win rates and rankings. Yes/No verification is used for binary judgments such as correctness or factual support. In open-ended settings, pairwise comparison has been especially common in benchmarks such as AlpacaEval, VicunaEval, and Arena-Hard, where evaluators compare alternative responses rather than attempting to assign calibrated absolute scores (Yu et al., 23 May 2025).

Human-centered systems have made these distinctions explicit. EvaluLLM organizes evaluation into Build, Review, and Inspect stages, using pairwise comparison and blind manual review to calibrate trust (Pan et al., 2024). EvalAssist supports both direct assessment and pairwise comparison, exposes task context and criteria in a portable format, and implements a prompt chain consisting of assessment, summarization, and answer selection (Ashktorab et al., 2 Jul 2025). A related study of practitioners found that direct assessment led to more evaluations, more task-specific criteria refinement, more judgment modification, and more evaluator-model switching than pairwise comparison (Ashktorab et al., 2024).

2. Reliability as the central technical problem

The core issue in LLM evaluation is not merely whether a model can produce plausible judgments, but whether those judgments are reliable across runs, languages, orderings, tasks, judges, and domains. The literature measures reliability through human agreement, inter-judge agreement, intra-judge stability, bias diagnostics, cross-lingual consistency, and task-specific constructs such as transitivity or abstention (Gu et al., 2024).

One influential reliability failure is non-transitive pairwise preference. In an ideal evaluator, if $A \succ B$ and $B \succ C$, then $A \succ C$. ELSPR models pairwise preferences as tournament graphs, uses strongly connected components to detect cycles, and introduces directed graph structural entropy as a measure of overall preference clarity (Yu et al., 23 May 2025). On this analysis, even advanced evaluator LLMs exhibit substantial cyclicity: Qwen2.5-Max shows 67.96% non-transitivity and 0.8095 average normalized structural entropy, indicating low preference clarity despite strong general capability (Yu et al., 23 May 2025).

A second major reliability axis is cross-lingual consistency. In a multilingual study spanning five model families, five tasks, and 25 languages, average Fleiss’ Kappa is reported at approximately 0.3, with especially poor performance in low-resource languages (Fu et al., 18 May 2025). On MGSM, Llama-3.3-70B reaches Cohen’s Kappa as low as 0.002 for Telugu versus English, showing that the same underlying content can receive sharply different judgments depending on language (Fu et al., 18 May 2025). The same study further shows that binary Yes/No judgments are more consistent than 1–5 grading (Fu et al., 18 May 2025).

A third line of evidence comes from work on evaluator heterogeneity. “Evaluative Fingerprints” reports Krippendorff’s $\mathcal{C}$0 across 3,240 evaluations from nine judges, with two rubric dimensions exhibiting $\mathcal{C}$1, yet also finds strong within-judge stability, with ICC values ranging from -0.038 to 0.872 (Nasser, 8 Jan 2026). The result is a reliability paradox: judges disagree with one another, but remain stable relative to themselves (Nasser, 8 Jan 2026).

Reliability difficulties are not unique to LLMs. In privacy evaluation, human raters themselves achieve only $\mathcal{C}$2 overall, with average pairwise human-human agreement of 0.54, indicating that some target constructs are intrinsically subjective (Meisenbacher et al., 16 Aug 2025). In that setting, LLMs align better with the average human rating per text than with individual human raters, suggesting that evaluator models may capture a global norm more readily than personal variation (Meisenbacher et al., 16 Aug 2025).

3. Bias, calibration, and the problem of what is being measured

The bias literature shows that evaluator outputs are shaped not only by content quality but also by presentation and model identity. The survey and human-centered design work catalogue position bias, verbosity bias, self-enhancement bias, prompt sensitivity, and ordering effects as recurring problems in LLM judges (Gu et al., 2024). MTDEval adds score compression, high variance, and leniency bias to this list in dialogue evaluation (Tang et al., 1 Aug 2025). In practice, many systems mitigate position bias by swapping answer order and checking whether the judgment changes (Pan et al., 2024).

Bias also appears at the model-family level. MedQADE studies nine LLM evaluators and finds that the top-performing evaluator, Gemini 3 Flash, reaches $\mathcal{C}$3, close to the physician ceiling of $\mathcal{C}$4, but still assigns a definitive score in every case, whereas physicians increase abstention as difficulty rises (Philipp et al., 1 Jul 2026). The same work reports lineage-dependent favoritism, including sibling-model advantages such as +6.63% for GPT-5.4 Mini scoring GPT-5 Nano and +11.54% for Gemma 3 4B scoring Gemma 3 27B (Philipp et al., 1 Jul 2026). This establishes that high statistical agreement does not imply human-like caution or independence.

A related analysis in merchant risk assessment introduces a judge-independent consensus-deviation metric and shows substantial evaluator heterogeneity. GPT-5.1 and Claude 4.5 Sonnet exhibit negative self-evaluation bias of -0.33 and -0.31, whereas Gemini-2.5 Pro and Grok 4 show positive bias of +0.77 and +0.71; anonymization reduces bias magnitude by 25.8 percent without reversing its sign (Wang et al., 4 Feb 2026). Human experts further show that LLM judges are, on average, +0.46 points more lenient than human consensus (Wang et al., 4 Feb 2026).

The applied classroom literature shows a more adversarial form of evaluator bias: vulnerability to prompt manipulation. In a 1,028-student course using GPT-4-turbo as an assignment evaluator, 51.3% of students reported output-format failures, 21.5% reported rubric-following failures with underscoring, and 47% admitted trying to prompt-hack the evaluator (Chiang et al., 2024). The same deployment documents direct command hijacks, rubric rewrites, score inflation, and post hoc self-reflective hacking detection (Chiang et al., 2024).

Two misconceptions are repeatedly rejected. First, accuracy and consistency are not the same: in multilingual judging, Spearman correlations between quality and Fleiss’ Kappa vary widely and can be zero or negative (Fu et al., 18 May 2025). Second, averaging judges does not necessarily recover a shared construct: “Evaluative Fingerprints” argues that averaging produces a synthetic verdict corresponding to no actual judge’s values (Nasser, 8 Jan 2026). This suggests that evaluator choice is not a neutral implementation detail but a methodological decision about what quality definition is being operationalized.

4. Methods for improving evaluator quality

The field has responded to reliability failures with interventions at the levels of prompt design, training data, aggregation, and evaluator architecture. The survey literature highlights few-shot prompting, step decomposition, criteria decomposition, repeated evaluation, candidate-order swapping, and pairwise ranking as general-purpose reliability strategies (Gu et al., 2024). Human-centered systems further emphasize blind review, prompt transparency, rationale inspection, and early evaluation on small subsets before scaling (Pan et al., 2024).

ELSPR addresses non-transitivity directly at the training-data level. It reconstructs pairwise preference data as tournament graphs, identifies strongly connected components with Tarjan’s algorithm, reconstructs each cyclic component into a DAG using in-degree as a win-score proxy, and retains only data points consistent with the purified transitive relations for fine-tuning (Yu et al., 23 May 2025). On averaged testing sets, training on Cleaned data reduces non-transitivity from 64.28% to 50.50%, lowers normalized structural entropy from 0.8113 to 0.7234, improves human agreement from 66.5% to 67.1%, and raises Spearman correlation from 0.96 to 0.97 (Yu et al., 23 May 2025). A random 20% removal baseline does not produce the same effect (Yu et al., 23 May 2025).

Aggregation is another major strategy. In multilingual judging, an ensemble of Llama-3.3-70B, Qwen-2.5-72B, and Aya-Expanse-32B uses majority vote to improve consistency over the weakest single judge, often by more than 0.1, though not uniformly (Fu et al., 18 May 2025). MTDEval replaces costly multi-judge inference with a single learned evaluator trained on about 11K dialogue pairs annotated by five state-of-the-art judges and optimized with a probabilistic ranking model based on Thurstone’s Case V and learned judge reliabilities (Tang et al., 1 Aug 2025). It is reported as the best open-source evaluator on seven of eight pairwise tasks and runs at 0.10 s / 0.19 s per instance on Daily-MTD / Daily-MTD-Pair (Tang et al., 1 Aug 2025).

Dynamic aggregation extends this idea further. LLM Jury-on-Demand trains judge-specific reliability predictors from token distributions, embeddings, and structural features, then selects an instance-specific top-$\mathcal{C}$5 jury and combines their scores with reliability-aware weights (Li et al., 1 Dec 2025). On summarization and RAG benchmarks, this dynamic jury achieves higher Kendall’s Tau with human judgments than both single-judge and static-jury baselines (Li et al., 1 Dec 2025).

Other work modifies the evaluator role itself. GER-Eval asks models to generate their own rubrics and then apply them; the generated rubrics typically contain five to eight criteria, are interpretable and task-aware, and yield agreement often in the 70–90% range with correlations frequently above 0.8, but degrade in factual and knowledge-intensive settings such as SumPubMed (Siro et al., 9 Feb 2026). MAJ-EVAL constructs evaluator personas and dimensions automatically from domain documents, then uses in-group debate among persona-conditioned LLM agents; in both education and medicine, it aligns better with human experts than ROUGE-L, BERTScore, G-Eval, and ChatEval on most task-specific dimensions (Chen et al., 28 Jul 2025).

5. Human-centered systems and domain-specific deployments

A distinctive feature of the literature is that evaluator design increasingly includes interface design, workflow design, and domain adaptation, not only judge prompting. EvaluLLM argues that LLM-as-a-judge should be interactive, blind, and iterative, with support for customizable criteria, pairwise comparison, rationales, and agreement displays rather than black-box automation (Pan et al., 2024). EvalAssist operationalizes this through an online criteria development environment, structured and portable criteria, positional-bias indicators, and export to UNITXT-backed notebooks; it has been deployed internally to over 700 users (Ashktorab et al., 2 Jul 2025).

The diversity of applications shows that “LLM as an evaluator” is not a single task family but a common evaluation architecture adapted to domain-specific rubrics, output formats, and failure modes.

Domain	Representative setting	Reported finding
ML explanations	GPT-4o and Mistral-7.2B judge LIME, similarity-based, and no-explanation conditions	LLMs assess subjective explanation quality reasonably well, but are not sufficiently developed to replace human judges (Wang et al., 28 Feb 2025)
Educational feedback	Fine-tuned GPT-4.1 “DeanLLM” over 16 feedback dimensions	Accuracy 79.8%, F1 79.4%; human average Accuracy 78.3%, F1 82.6% (Qian et al., 8 Aug 2025)
Classroom assignments	GPT-4-turbo “LLM TA” in a 1,028-student course	75% found use acceptable under proper settings; 51.3% reported format failures; 47% tried prompt-hacking (Chiang et al., 2024)
Clinical note consistency	Llama2-70B evaluates age, gender, body part, and coherence	Cohen’s kappa with consensus human reaches 0.79 for age and 1.00 for gender in 2-shot, but coherence remains unstable (Brake et al., 2024)
Privacy sensitivity	13 LLM judges rate 250 texts on a 1–5 privacy scale	LLMs approximate a global human privacy perspective, while human agreement itself remains low (Meisenbacher et al., 16 Aug 2025)

These deployments show a recurring pattern. LLM evaluators work best when the target construct is explicit, the rubric is operationalized carefully, and the evaluation remains low enough stakes for human correction or oversight. They are less stable when the task depends on ambiguous social norms, adversarial interaction, or latent domain knowledge that is not fully captured by the prompt (Wang et al., 28 Feb 2025).

6. Conceptual implications and open directions

The literature increasingly treats evaluator capability as a distinct competency rather than a by-product of generation quality. Several findings support this separation. Qwen2.5-Max is a strong model yet exhibits high non-transitivity as a judge (Yu et al., 23 May 2025). Multilingual training and larger model scale do not directly improve cross-lingual judge consistency (Fu et al., 18 May 2025). Fine-tuned evaluator models can exceed stronger general-purpose judges on specific clarity or agreement metrics (Yu et al., 23 May 2025).

A broader implication is that evaluator behavior appears to be internally coherent but externally fragmented. GER-Eval finds that models can design interpretable, task-aware rubrics and apply them consistently within-model, while cross-model generalization weakens in knowledge-intensive settings (Siro et al., 9 Feb 2026). “Evaluative Fingerprints” similarly finds that evaluators are stable enough to be identified from rubric scores alone with 77.1% accuracy, rising to 89.9% with disposition features, and that even GPT-4.1 and GPT-5.2 can be distinguished with 99.6% accuracy within-family (Nasser, 8 Jan 2026). This suggests that different judges instantiate different evaluative dispositions rather than noisy estimates of a single latent construct.

For deployment, the strongest consensus is procedural. Trustworthy evaluator systems require explicit bias auditing, prompt and rubric transparency, order swapping, judge-independence checks, domain-specific calibration, and human-in-the-loop validation (Pan et al., 2024). In high-stakes settings, statistical alignment alone is insufficient: MedQADE argues that evaluator independence and uncertainty behavior must be verified explicitly, not inferred from kappa alone (Philipp et al., 1 Jul 2026). The survey literature therefore frames the central question not as whether LLMs can judge, but how to build reliable LLM-as-a-Judge systems through consistency controls, debiasing, scenario adaptation, and principled meta-evaluation (Gu et al., 2024).

The field’s present state is therefore dual. LLM evaluators are already practical for ranking, filtering, rubric-based screening, and scalable assistance across many domains. Yet the accumulated evidence also shows that they are not interchangeable instruments, that agreement with humans is only one component of validity, and that evaluator design now includes graph structure, aggregation theory, interface design, and domain governance as much as prompt engineering.