Judge Q: LLM Evaluation & KV Cache Optimization

• Judge Q is a mechanism that transforms raw LLM assessments into calibrated, human-aligned scores using pairwise and pointwise evaluation methods.
• Studies reveal that Judge Q evaluations are sensitive to prompt wording, language switching, and positional bias, necessitating multi-trial aggregation and rigorous calibration.
• In a separate context, Judge Q refers to trainable query tokens for KV cache eviction that improve information retention and performance under tight memory constraints.

In recent arXiv usage, “Judge Q” appears in at least two senses. Most commonly, it denotes an LLM-based evaluator or a calibrated secondary model that transforms judge rationales and raw scores into human-aligned quantitative predictions; in a separate long-context systems line, it is the title of “Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction,” where “Judge Q” refers to trainable query tokens that score KV-cache importance (Sahoo et al., 3 Jun 2025, Liu et al., 13 Sep 2025). Across the evaluator literature, the shared concern is whether automated judgments preserve the target signal—human preference, correctness, safety, or usefulness—without introducing uncontrolled noise, shortcut bias, or calibration error (Yagubyan, 23 Apr 2026).

1. Conceptual scope and formalization

In the LLM-as-a-judge setting, a LLM is used purely as an evaluator: given a task description and one or more candidate answers, it returns either a scalar score or a preference among answers. One formalization treats a judge evaluation trial as a mapping

$$(q, r_A, r_B, p, \theta) \rightarrow \text{output},$$

where $q$ is the user question or task, $r_A$ and $r_B$ are candidate responses, $p$ is the evaluation prompt template, and $\theta$ denotes judge settings such as model choice, temperature, and response order. Two standard modes are distinguished: pairwise judging, where the output is $y \in \{A, B, \text{tie}\}$, and pointwise judging, where each response receives $s \in [1,10]$ (Yagubyan, 23 Apr 2026).

A closely related formulation separates absolute and relative judges. In the absolute case, the judge maps a prompt–response pair $(x,y)$ to a textual evaluation $e$ and numerical score $q$0. In the relative case, it maps $q$1 to a textual comparison $q$2 and binary preference $q$3. “Quantitative LLM judges” then place a post-hoc GLM on top of the frozen judge outputs, using some combination of the textual embedding $q$4, the base score $q$5, and base probabilities $q$6, to predict calibrated human-aligned scores or preferences. The four explicit variants are the Least-Squares judge, the Multinomial judge, the Bradley–Terry–Luce judge, and the two-headed BTL2 judge (Sahoo et al., 3 Jun 2025).

This suggests a useful distinction between raw judges and calibrated judges. Raw judges emit direct preferences or scores from a prompting protocol. Calibrated judges preserve the frozen judge’s qualitative reasoning while learning a low-dimensional quantitative map to human labels. The literature also extends the judge role beyond text-only evaluation: multimodal judges score or rank image-grounded answers, and document-grounded multi-turn judges must identify not only which conversation is worse, but also the flawed turn and the failure type (Chen et al., 2024, Tang et al., 20 May 2026).

2. Reliability, instability, and hidden bias

Run-to-run reliability is now one of the central questions in Judge Q research. Repeated identical evaluations on 29 tasks spanning 10 categories, using GPT-4o-mini and GPT-4.1-mini as judges, found that pairwise preferences flip on average 13.6% of the time; 28% of questions exceed a 20% flip rate; and one GPT-4.1-mini reasoning item reaches 56%. The paper defines flip rate as

$q$7

the fraction of trials not supporting the majority verdict. The same study also reports a statistically significant first-position bias for GPT-4o-mini, with A-majority on 21 of 29 questions, or 72%, $q$8; a pairwise–pointwise gap in which mean pointwise score gaps are only 0.19–0.36 on a 10-point scale and not statistically significant in aggregate; cross-judge agreement of only 76% with $q$9; semantically equivalent prompt templates changing majority outcomes in 25% of tested cases; and a reliability-curve estimate that 11 repeated trials are needed, on average, for a majority vote to recover the 50-trial reference verdict with 95% probability, rising to 15 for high-variance questions (Yagubyan, 23 Apr 2026).

A second line of work tests whether judges rely on irrelevant metadata shortcuts. Using controlled cue perturbations on ELI5 and LitBench, the paper measures Verdict Shift Rate (VSR) and Cue Acknowledgment Rate (CAR) for cue families including source, temporal, age, gender, ethnicity, and educational status. Across cues with strong behavioral effects, including provenance hierarchies such as Expert $r_A$0 Human $r_A$1 LLM $r_A$2 Unknown, recency preferences $r_A$3, and educational-status favoritism, CAR is “typically at or near zero,” even when VSR is large. On LitBench, for example, Claude-3-Haiku shows temporal VSR = 71% and CAR = 0%, while Qwen3-235B shows educational-status VSR = 74% and CAR = 13%. This combination of substantial verdict sensitivity and limited cue acknowledgment is described as an explanation gap in LLM-as-judge pipelines (Marioriyad et al., 8 Feb 2026).

Language-switching adds another invariance failure mode. Judge-LS evaluates four API-accessible judges on all 419 LLMBar items under English, Chinese, and Chinese–English language-switched variants, producing 13,408 successful pairwise judgments. Across models, Chinese and language-switched presentations induce 10.7–14.4% preference flips relative to English, and all judges achieve their highest accuracy in English. At the same time, translation-equivalent tie probes do not show a systematic English preference: most probes are judged as ties, and among non-tie decisions Chinese wins more often than English for every model (Yin, 12 Jun 2026).

Taken together, these results show that Judge Q is not merely noisy in the classical statistical sense. It is sensitive to response order, prompt wording, language presentation, and injected metadata, and its natural-language justifications often do not expose those dependencies. This suggests that invariance cannot be assumed merely because the underlying task or answers are held fixed.

3. Domain-specific benchmarks and judge architectures

Benchmark construction has become a major research theme because judge behavior is highly domain-dependent. In coding, CodeJudgeBench introduces 4,260 curated pairs across code generation, code repair, and unit test generation. It finds that recent thinking models substantially outperform non-thinking models; even relatively small thinking models such as Qwen3-8B outperform specially trained LLM-as-a-Judge models up to 70B in size; all models still exhibit significant randomness; changing answer order can substantially impact accuracy; pairwise comparison outperforms scalar pointwise judging; and retaining comments and reasoning in the full, unprocessed response improves judge performance (Jiang et al., 14 Jul 2025).

In safety evaluation, OBJEX(MT) isolates objective extraction in multi-turn jailbreaks. Each instance requires a judge to output a JSON object with a single-sentence base_prompt and a confidence score in $r_A$4. Using a human-aligned threshold $r_A$5, claude-sonnet-4 attains the highest objective-extraction accuracy at 0.515 and the best calibration with ECE 0.296 and Brier 0.324, while gpt-4.1 and Qwen3 tie at 0.441 accuracy and remain markedly overconfident, with mean confidence approximately 0.88 versus accuracy approximately 0.44 and [email protected] approximately 48–52%. Performance varies sharply across datasets, from approximately 0.167 to 0.865, with MHJ easiest and Attack_600 and CoSafe harder (Kim et al., 23 Aug 2025).

In financial discourse, EvasionBench builds a three-class evasion benchmark for earnings-call Q&A with 30,000 training samples and 1,000 human-annotated test samples, reporting Cohen’s Kappa 0.835 on the human test annotations. Its annotation pipeline uses Claude Opus 4.5 and Gemini-3-Flash as dual annotators and invokes a judge on disagreement cases. The resulting 4B detector, Eva-4B, reaches 81.3% accuracy and F1-macro 0.807, outperforming its base model by 25.1 percentage points in accuracy and improving by 2.4 percentage points over an Opus-only single-teacher baseline (Ma et al., 14 Jan 2026).

Multi-turn, document-grounded judging is addressed by RankJudge, which generates pairs of conversations where one conversation contains exactly one injected flaw in a specific assistant turn. The judge must jointly identify the better conversation, the flawed round, and the failure type. RankJudge evaluates 21 frontier judges and ranks them via the Bradley–Terry model; on the curated 652-pair slice, gemini-3.1-pro leads at $r_A$6 Elo, followed by gpt-5.5 at $r_A$7 (Tang et al., 20 May 2026).

Multimodal evaluation has produced two distinct strands. “MLLM-as-a-Judge” shows that pair comparison is the most human-aligned multimodal judging task: GPT-4V reaches average pair accuracy 0.683 with tie and 0.806 without tie, whereas scoring and batch ranking diverge more sharply from human judgments and exhibit biases, hallucinations, and inconsistencies. “MR. Judge” instead reformulates multimodal judging as a reasoning-based multiple-choice problem over candidate responses, with a deliberate > phase followed by boxed{A/B/C/D} selection. MR. Judge-7B-SFT-RL reaches 75.5 overall on VL-RewardBench, surpassing GPT-4o at 65.8, and improves MM-Vet by up to 7.7 points during inference-time scaling (Chen et al., 2024, Pi et al., 19 May 2025).

Even relatively classical extractive QA now shows the same pattern. On Quoref, DROP, HotpotQA, and 2WikiMultiHopQA, LLM-as-a-judge reaches average Pearson correlation 0.847 with human labels, compared with 0.174 for EM and 0.359 for F1. The best judge in that study, Qwen 2.5 72B, shows especially strong agreement for numbers and dates, while job-type answers remain harder; the paper also reports no serious self-preference issue in this extractive setting (Ho et al., 16 Apr 2025).

4. Calibration, correction, and uncertainty quantification

A raw judged correctness rate is not, in general, an estimate of true human correctness. If $r_A$8 is the human label, $r_A$9 is the judge label, $r_B$0 is the true accuracy, $r_B$1 is judge sensitivity, and $r_B$2 is judge specificity, then the judged-correct rate satisfies

$r_B$3

Accordingly, the naive estimator $r_B$4 is biased for $r_B$5 unless the judge is perfect. A plug-in correction uses a calibration set with both human and judge labels and estimates

$r_B$6

The same work develops adjusted confidence intervals that incorporate uncertainty from both the test set and the calibration set, and proposes an adaptive calibration-allocation rule to reduce interval width (Lee et al., 26 Nov 2025).

For model comparisons, the situation is more delicate. “Bias and Uncertainty in LLM-as-a-Judge Estimation” introduces judge quality

$r_B$7

and cross-model calibration instability

$r_B$8

as diagnostics. Low $r_B$9 means the judge is close to random; nonzero $p$0 means the judge behaves differently on different base models. The paper shows analytically and in simulation that shared calibration across compared models can introduce severe bias, and in a real MMLU-Pro case study it produces sign reversal: naive and shared-calibration corrected estimates can claim the wrong model is better with high apparent confidence. Because shared-calibration comparison estimators effectively divide by $p$1, both variance and bias can be amplified when judge quality is poor (Fiedler, 7 May 2026).

A separate response to calibration is not to model judge misclassification directly, but to align raw judge outputs to human scores post hoc. In “Quantitative LLM Judges,” the base judge first emits a rationale $p$2 and score or probability, and a small GLM then predicts the calibrated output from $p$3, $p$4, and $p$5. On Summarize from Feedback, for example, a Prometheus base judge has MSE 6.346, while the Least-Squares quantitative judge reduces that to 2.626. On Offset Bias, BTL2 reaches accuracy 0.783 and Pearson $p$6, outperforming the raw relative base and remaining competitive with supervised fine-tuning while using far less compute (Sahoo et al., 3 Jun 2025).

These strands address different layers of the same problem. Statistical correction treats the judge as a noisy measurement device whose sensitivity and specificity must be estimated. Quantitative judging treats the judge as a frozen qualitative reasoner whose outputs can be calibrated by a secondary predictor. Both reject the assumption that raw judge outputs are self-justifying.

5. Operational doctrine for deploying Judge Q

The strongest operational consensus in the literature is that single-pass judging should not be treated as a gold-standard measurement. For competitive pairwise evaluations, the reliability study on GPT-4o-mini and GPT-4.1-mini recommends multi-trial aggregation, position randomization, prompt standardization, explicit uncertainty reporting, and multiple judges. A practical rule of thumb from that work is 3–5 trials for rough comparisons, approximately 10–20 for publication-level comparisons, and about 11 trials on average for 95% stability relative to a 50-trial reference verdict, rising to 15 or more for hard items (Yagubyan, 23 Apr 2026).

Where the judge must infer latent objectives, the recommendation changes. OBJEX(MT) argues that judges should not be assumed to reliably reconstruct hidden goals from long, noisy, adversarial dialogues. When feasible, the evaluation objective should be provided explicitly, and confidence should be treated as a noisy signal rather than a safe abstention criterion: for gpt-4.1 and Qwen3 in that benchmark, confidence $p$7 still corresponds to roughly half the predictions being wrong. The paper therefore recommends selective prediction or abstention, but only after calibration on deployment-like data (Kim et al., 23 Aug 2025).

For supervised-data construction, disagreement itself can be operationally useful. EvasionBench treats disagreement between frontier annotators as a hard-example signal, routes those cases to a judge, and uses the resulting labels to train a smaller specialist model. The reported training-loss pattern—0.421 for the multi-model-plus-judge training set versus 0.393 for the Opus-only baseline, yet better generalization—supports the view that disagreement mining acts as implicit regularization (Ma et al., 14 Jan 2026).

For multilingual evaluation, Judge-LS suggests two pragmatic options: normalize to English when translation quality is trustworthy, or evaluate in multiple language conditions and report language-switching flip rates, position inconsistency, and tie-probe behavior. For any setting in which the judge, prompt, task domain, or provider configuration changes, the reporting literature recommends recalibration rather than reusing stale judge error rates (Yin, 12 Jun 2026, Lee et al., 26 Nov 2025).

This suggests that Judge Q should be treated as a measurement pipeline rather than a single model call. The pipeline includes prompt choice, randomization, repeated sampling, calibration data, cross-model stability checks, and uncertainty reporting. On that view, the judge is not a substitute for methodology; it is one component inside it.

6. Judge Q as trainable queries for KV-cache eviction

A separate systems paper uses the same name for a problem unrelated to model evaluation. In “Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction,” Judge Q refers to a trainable soft-token list appended to the prompt during prefill so that the resulting queries can score which key–value entries should be retained under a fixed KV budget. Standard eviction methods often use the last window from the prefill stage as queries, which the paper argues overly focuses on local information. Judge Q instead trains only the model’s embedding layer for these soft tokens, leaving the rest of the model frozen, and aligns the average soft-token attention map

$p$8

to the average response-token attention map

$p$9

with an MSE loss (Liu et al., 13 Sep 2025).

The default configuration uses $\theta$0 soft tokens, trains on 50,000 ShareGPT-derived examples, and updates only the soft-token embeddings with AdamW for 3 epochs at learning rate $\theta$1 on a single A100 40GB GPU. At inference, the prompt is concatenated with the soft tokens, their attention to the prompt is aggregated into token-importance scores, the top-$\theta$2 KV entries are retained, and the soft tokens are then discarded before normal decoding (Liu et al., 13 Sep 2025).

Under equal eviction budgets, the method shows smaller degradation than StreamingLLM, H2O, SnapKV, and PyramidKV. On LongBench with Llama-3.1-8B-Instruct and budget 128, Judge Q reaches 35.90 versus 34.31 for the best baseline; with Mistral-7B-Instruct-v0.3 at the same budget it reaches 42.38 versus 39.75. On RULER, the gains are larger: with Llama-3.1-8B at sequence length 8192 and budget 256, Judge Q reaches 63.13 versus 57.83 for the best baseline; with Mistral-7B at the same setting, it reaches 52.72 versus 42.44. The paper summarizes the overall effect as approximately 1 point on LongBench and over 3 points on RULER, with stronger gains at tighter budgets and on retrieval-style settings such as Needle-in-a-Haystack (Liu et al., 13 Sep 2025).

In this systems sense, Judge Q does not evaluate answer quality or human preference. It evaluates KV importance. The shared label is nevertheless conceptually consistent: in both the evaluator literature and the KV-eviction paper, Judge Q is a mechanism for prioritizing uncertain alternatives under constrained observation.