Insights into Judge Response Theory

• Judge Response Theory is a framework that defines evaluation as a non-neutral process shaped by normative commitments, stochasticity, and judge-specific decision logic.
• It employs comparative analysis—using pairwise rankings, latent score models, and statistical generalizations—to reveal systematic discrepancies between human and LLM judges.
• Extensions such as self-judging, meta-judging, and multimodal transfer offer practical methods to improve AI safety, alignment, and scalable automated evaluation.

Judge Response Theory denotes a family of frameworks for analyzing how model outputs are evaluated by humans, by other LLMs, or by hybrid panels, with the central claim that judging is not a neutral descriptive act but an operation shaped by normative commitments, stochasticity, and judge-specific decision logic. In the formulation derived from Pasch’s study of content moderation, the theory treats “Human Judges” and “LLM Judges” as distinct “judge types” on a spectrum with different value priors, and uses refusal responses as a particularly revealing test case for divergence in evaluation (Pasch, 21 May 2025). In subsequent work, the label is extended to stochastic budgeted evaluation, meta-judging, cross-query reliability ranking, multimodal reasoning-guided judging, and heterogeneous multi-judge ranking, thereby broadening the theory from a content-moderation analysis into a general account of comparative judgment in automated evaluation (Saha et al., 17 Feb 2026, Wu et al., 2024, Liu et al., 2024, Ko et al., 24 May 2025, Yu et al., 6 May 2026).

1. Conceptual basis and judge types

Judge Response Theory begins from the observation that evaluating an AI output is “not a neutral, purely descriptive act.” Whether the judge is a human rater or an LLM, preferring one response over another enacts norms about tone, helpfulness, risk, and moral appropriateness (Pasch, 21 May 2025). In present practice, “LLM-as-a-Judge” frameworks are described as a staple of “large-scale benchmarking, fine-tuning pipelines (e.g., RLHF, DPO), and safety audits,” valued for “consistency and scalability” but already associated with documented biases such as favoring verbosity or emotional neutrality (Pasch, 21 May 2025).

Within this frame, judge types are differentiated by their value priors. Human judges are said to internalize “interpersonal norms—cooperation, conversational flow, and expectations of responsiveness,” whereas LLM judges reflect “the objectives embedded during training (e.g., developer-specified safety and alignment priorities)” (Pasch, 21 May 2025). The theory’s primary explanatory ambition is therefore comparative: it seeks to explain when human and model-based evaluators agree, when they diverge, and what that divergence reveals about alignment objectives embedded in evaluation systems.

A later statistical generalization appears in Heterogeneous Judge-Aware ranking, which formalizes the idea that multi-judge data should not be reduced to a single pooled score that treats disagreement as noise. Instead, it separates “consensus ranking, judge-specific sensitivity to consensus, and residual preference disagreement” as distinct inferential targets (Yu et al., 6 May 2026). This pushes Judge Response Theory toward a response-theoretic view in which judge heterogeneity is intrinsic structure rather than annotation error.

2. Refusals as the canonical object of analysis

In the content-moderation formulation, the theory operationalizes refusal behavior along two dimensions: motivation and form (Pasch, 21 May 2025). Under motivation, “ethical refusals” are refusals grounded in “normative or safety concerns,” such as harmfulness or illegality, whereas “technical refusals” are refusals grounded in capability limitations, such as lacking real-time data or expertise. Under form, the theory distinguishes “Full Refusals,” where the model offers no substantive answer, from “Disclaimers,” where the model states limits but still attempts a partial answer (Pasch, 21 May 2025).

Pasch’s study operationalized these distinctions by human-annotating 3,500 responses and training “a RoBERTa classifier (88% accuracy/F1) to label all pairs in Chatbot Arena,” producing five categories: “ethical refusal, ethical disclaimer, technical refusal, technical disclaimer, and standard” (Pasch, 21 May 2025). These categories are theoretically diagnostic. Ethical refusals are presented as a test of “alignment bias”: they reveal whether judges reward a response because it demonstrates value-guided constraint. Technical refusals test whether judges reward transparency and humility about system limitations (Pasch, 21 May 2025).

This typology is central because refusal responses are simultaneously safety behavior, conversational behavior, and evaluative stimuli. A refusal can signal alignment from one perspective and uncooperativeness from another. Judge Response Theory therefore treats refusals as a case in which the normative assumptions of the judge are especially visible.

3. Empirical divergence and “moderation bias”

The core empirical setting is Chatbot Arena, using “49,938 one-turn prompt / response-pair comparisons after filtering out multi-turn examples,” where human users select “Win/Loss/Tie” for response A versus B (Pasch, 21 May 2025). The same pairwise format was used with two LLM judges, “GPT-4o” and “Llama 3 70B,” under a “Standardized pairwise-comparison template,” with controls for “Response length (Z-scored), cosine similarity between prompt and response (all-MiniLM-L6-v2 embeddings, Z-scored),” and opponent-model features included in regression (Pasch, 21 May 2025).

The central finding is a divergence between human judges and LLM judges for ethical refusals. For ethical refusals, the reported win rates are $\mu_{\text{human}} = 0.08$, $\mu_{\text{GPT}} = 0.31$, and $\mu_{\text{Llama}} = 0.27$, yielding $\Delta \mu_{\text{GPT-human}} = +0.23$ and $\Delta \mu_{\text{Llama-human}} = +0.19$ (Pasch, 21 May 2025). For technical refusals, the divergence is smaller: $\mu_{\text{human}} = 0.16$, $\mu_{\text{GPT}} = 0.27$, and $\mu_{\text{Llama}} = 0.24$, with $\Delta \mu_{\text{GPT-human}} = +0.11$ and $\Delta \mu_{\text{Llama-human}} = +0.08$ (Pasch, 21 May 2025). For standard responses, the summaries report that “GPT and human judges roughly align” (Pasch, 21 May 2025).

Regression estimates sharpen the contrast. For ethical refusal, the OLS coefficient on “Refusal Ethical” for Win is $\mu_{\text{GPT}} = 0.31$0 for human judges, $\mu_{\text{GPT}} = 0.31$1 for GPT-4o judges, and $\mu_{\text{GPT}} = 0.31$2 for Llama 70B judges, with $\mu_{\text{GPT}} = 0.31$3 and all $\mu_{\text{GPT}} = 0.31$4 (Pasch, 21 May 2025). For technical refusals, GPT-4o’s Win coefficient of $\mu_{\text{GPT}} = 0.31$5 “does not differ significantly from humans” at $\mu_{\text{GPT}} = 0.31$6 with $\mu_{\text{GPT}} = 0.31$7, while Llama penalizes more severely at $\mu_{\text{GPT}} = 0.31$8 with $\mu_{\text{GPT}} = 0.31$9 (Pasch, 21 May 2025). The effect-size summary states that “ethical refusals are ~4× more ‘rewarded’ by AI judges relative to humans” (Pasch, 21 May 2025).

Pasch names this divergence “moderation bias,” defined as “a systematic tendency for model-based evaluators to reward refusal behaviors more than human users do” (Pasch, 21 May 2025). The asymmetry matters: AI judges reward ethical refusals more strongly, but technical refusals are treated comparably to humans or, in Llama’s case, more harshly. Human judges, by contrast, exhibit a “refusal penalty,” with ethical refusals seen as “moralizing or evasive” and technical refusals only “mildly less penalized” (Pasch, 21 May 2025). This suggests that alignment-sensitive evaluators can embed safety priorities that diverge from user preference as ordinarily expressed in pairwise choice data.

4. Formal models of judging: stochasticity, ranking, and identifiability

A separate line of work formalizes LLM judging as a stochastic estimation problem under a fixed query budget. In “LLM-as-Judge on a Budget,” there are $\mu_{\text{Llama}} = 0.27$0 fixed prompt–response pairs with unknown scores $\mu_{\text{Llama}} = 0.27$1, and each judgment is modeled as

$\mu_{\text{Llama}} = 0.27$2

with zero-mean noise and variance bounded by $\mu_{\text{Llama}} = 0.27$3 (Saha et al., 17 Feb 2026). Given a total budget $\mu_{\text{Llama}} = 0.27$4, the objective is to minimize the expected worst-case estimation error over sample-mean estimates. In the known-variance setting, the minimax allocation is

$\mu_{\text{Llama}} = 0.27$5

and the paper proposes a variance-adaptive algorithm for the unknown-variance case using an exploration phase and an upper-confidence-bound estimate of each variance (Saha et al., 17 Feb 2026). The resulting error rate is summarized as

$\mu_{\text{Llama}} = 0.27$6

with practical significance for “AI safety, model alignment, and automated assessment at scale” (Saha et al., 17 Feb 2026).

A different formalization appears in Heterogeneous Judge-Aware ranking, where pairwise judgments are modeled through a latent score matrix $\mu_{\text{Llama}} = 0.27$7 and Bradley–Terry–Luce log-odds:

$\mu_{\text{Llama}} = 0.27$8

The key decomposition is

$\mu_{\text{Llama}} = 0.27$9

where $\Delta \mu_{\text{GPT-human}} = +0.23$0 is the consensus ranking direction, $\Delta \mu_{\text{GPT-human}} = +0.23$1 is judge $\Delta \mu_{\text{GPT-human}} = +0.23$2’s “consensus sensitivity,” and $\Delta \mu_{\text{GPT-human}} = +0.23$3 is a low-rank residual capturing structured disagreement (Yu et al., 6 May 2026). The paper imposes centering, scaling, orthogonality, and subspace anchoring conditions to identify $\Delta \mu_{\text{GPT-human}} = +0.23$4, and develops an “anchored alternating-block MLE algorithm” with re-anchoring to preserve the identifying geometry (Yu et al., 6 May 2026). Under a fixed-panel repeated-comparison regime, it derives Wald confidence intervals for consensus contrasts, sensitivity parameters, pairwise probabilities, and summaries of residual disagreement.

These two mathematical lines are complementary. The budgeted framework treats the judge as a stochastic oracle with unknown but bounded variance, whereas HJA treats multi-judge structure as a latent comparative system with identifiable heterogeneity. Together they support a broadened Judge Response Theory in which evaluation error arises both from stochastic variability and from systematic variation in judge sensitivity and disagreement.

5. Self-judging, meta-judging, and cross-query comparison

Judge Response Theory is also extended to self-improving evaluation. In “Meta-Rewarding LLMs,” a single model can play “Actor, Judge, and Meta-Judge,” with a primary reward function

$\Delta \mu_{\text{GPT-human}} = +0.23$5

and Direct Preference Optimization used both for improving responses and improving judgments (Wu et al., 2024). For judgment improvement, the model collects multiple independent judgments $\Delta \mu_{\text{GPT-human}} = +0.23$6 on a single response, compares the judgments pairwise via a Meta-Judge, solves for Elo scores, and trains on the highest-Elo versus lowest-Elo judgment pair (Wu et al., 2024). Empirically, starting from Llama-3-8B-Instruct, the reported win rate improves “from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard” (Wu et al., 2024). The theoretical significance is that judge quality itself becomes a trainable object rather than a fixed supervisory channel.

“Meta Ranking” addresses a different problem: single-response reliability judgment by comparatively weak LLMs. It defines a target query–response pair and compares it against a small labeled reference set through pairwise cross-query comparisons, with outputs in $\Delta \mu_{\text{GPT-human}} = +0.23$7 and an aggregation rule based on vote increments (Liu et al., 2024). The paper reports that “MR with Phi-2 achieves precision ≈0.77 on MMLU—double P(True) with the same model and 88% of GPT-4-turbo,” using as few as five reference samples (Liu et al., 2024). It also reports downstream improvements in model cascading and iterative instruction tuning, including routing from “OpenChat-3.5→GPT-4” with accuracy “≈52%→64%” at “token cost ratio 0.43” (Liu et al., 2024).

Both methods expand the scope of Judge Response Theory beyond direct pairwise answer comparison. Meta-Rewarding adds second-order judgment of judgments, while Meta Ranking replaces same-query comparison with cross-query comparison against external references. A plausible implication is that “judging” can be decomposed not only by judge type, but also by supervision topology: direct evaluation, self-evaluation, meta-evaluation, and reference-based comparison.

6. Multimodal transfer, adversarial vulnerability, and open issues

Flex-Judge extends the theory into multimodal evaluation by advancing the hypothesis that explicit textual reasoning captures “abstract decision-making primitives (correctness, completeness, coherence, relevance, etc.)” that are independent of modality (Ko et al., 24 May 2025). The model fine-tunes only the language transformer of an MLLM on pure text prompts with structured > …<answer>…</answer> supervision, while leaving the vision, audio, or molecule encoder and cross-attention bridge frozen (Ko et al., 24 May 2025). The paper presents this as “reasoning-based text supervision” that can transfer to “text, image, audio, and molecular tasks,” and describes the reasoning traces as an interpretable “bottleneck” for valuation criteria (Ko et al., 24 May 2025). This suggests a modality-agnostic extension of Judge Response Theory in which the core judging function resides in language-mediated decision structure rather than in modality-specific supervision.

At the same time, the literature highlights substantial vulnerabilities. “Optimization-based Prompt Injection Attack to LLM-as-a-Judge” formulates adversarial manipulation of a judge as a discrete optimization problem over an appended token sequence $\Delta \mu_{\text{GPT-human}} = +0.23$8, minimizing a composite loss built from “target-aligned generation loss,” “target-enhancement loss,” and “adversarial-perplexity loss” (Shi et al., 2024). The key mechanism is that if the judge makes its decision by generating a textual choice, then maximizing the probability of the target-selection string can subvert the decision rule itself. The paper reports, for example, “ASR ≈89.2%, PAC≈79%” on MT-Bench with OpenChat-3.5 and “ASR≈93.2%, PAC≈86.6%” on LLMBar with Mistral-7B (Shi et al., 2024). This establishes that judge outputs can be strategically steered even when baseline accuracy is high.

Several open issues recur across the literature. Pasch emphasizes “Hidden Value Priors” and “Feedback-Loop Risk,” warning that when LaaJ evaluations feed back into training or model selection, they can “systematically favor ethics-driven refusals, potentially at the expense of user satisfaction or collaborative norms” (Pasch, 21 May 2025). The budgeted-evaluation framework explicitly notes that it “do[es] not model dependencies across pairs or time-varying judge behavior (distribution shift or model updates)” (Saha et al., 17 Feb 2026). HJA argues that standard pipelines often collapse systematic disagreement into noise, particularly problematic in “near-tie regimes” where uncertainty calibration matters (Yu et al., 6 May 2026). The Pasch exposition also recommends “Transparent ‘Evaluation Cards’,” “Human-in-the-Loop Checks,” “Participatory Norm Definition,” and “Multi-Metric Evaluation” rather than sole reliance on pairwise Win/Loss/Tie signals (Pasch, 21 May 2025).

One additional point is a documentary inconsistency within the available account of JudgeDeceiver. The abstract states that the paper considers “known-answer detection, perplexity detection, and perplexity windowed detection,” while the detailed exposition states that it “does not propose or evaluate any defenses” (Shi et al., 2024). The discrepancy does not alter the paper’s central role in the theory, but it illustrates a broader methodological concern already foregrounded by Judge Response Theory itself: evaluation systems and reports are normatively and operationally loaded objects that require explicit auditing.

7. Position within evaluation theory

Judge Response Theory stands at the intersection of content moderation research, reward modeling, pairwise preference learning, multi-armed bandits, reliability estimation, and response theory. In its narrowest formulation, it explains why LLM judges and human users can assign systematically different value to the same refusal behavior (Pasch, 21 May 2025). In its broader mathematical and methodological formulations, it treats judges as stochastic oracles, as heterogeneous panel members with identifiable sensitivity parameters, as self-improving meta-evaluators, and as multimodal reasoning systems (Saha et al., 17 Feb 2026, Yu et al., 6 May 2026, Wu et al., 2024, Ko et al., 24 May 2025).

Relative to classical Item-Response Theory, the HJA exposition makes the connection explicit: “Classical 2-PL IRT is a special case of JRT with $\Delta \mu_{\text{GPT-human}} = +0.23$9,” while the more general model adds a low-rank residual to capture “systematic, multidimensional disagreement” orthogonal to consensus (Yu et al., 6 May 2026). This reframing is significant because it shifts the interpretation of disagreement. Rather than treating divergent judgments as annotation noise, Judge Response Theory treats them as evidence about value priors, sensitivity to consensus, and structured affinity patterns.

The unifying claim across the literature is therefore not that one judge is intrinsically correct, but that every judging procedure embeds assumptions about what counts as a good response. In the moderation setting, those assumptions appear as differential reward for ethical versus technical refusals. In self-rewarding systems, they appear as learned reward models and meta-judgments. In multimodal judging, they appear as reasoning bottlenecks that encode transferable decision criteria. In multi-judge ranking, they appear as decomposable consensus, sensitivity, and disagreement. Judge Response Theory, taken in this broad sense, is the study of those assumptions as first-class objects of empirical measurement, formal modeling, and system design.