Q-Judger: Advanced Query-Output Evaluator

• Q-Judger is a framework for query–output evaluation that uses reference-free scoring, pairwise comparisons, and structured critiques to assess candidate responses.
• It integrates methodologies like decentralized Proof-of-Quality, verifiable solution validation, and multi-turn conversation benchmarking to ensure robust evaluation.
• The system employs various architectures (e.g., TextCNN, MiniLM, DeBERTa) and extends to multimodal, cohort-aware, and psychometric regimes for comprehensive quality measurement.

In recent research, Q-Judger appears in overlapping ways as a label for judge systems that evaluate a query together with one or more candidate outputs: as a reference-free quality evaluator for decentralized inference, as a discriminative validator for candidate solution responses with verifiable answers, and as a benchmarking protocol whose single-turn assumptions are explicitly generalized to multi-turn, document-grounded interaction (Tian et al., 20 Apr 2026, Duo et al., 13 Jan 2026, Tang et al., 20 May 2026). Across these settings, the judge is not merely a scalar scorer. It may emit a real-valued score, a pairwise preference, a binary correctness verdict, a critique plus verdict token, or a structured triple that localizes an error and classifies its type. The resulting research program treats judging as a first-class modeling problem, with its own architectures, supervision signals, benchmarks, psychometric failure modes, and mechanistic structure.

1. Core formulations

A central formulation casts Q-Judger as a query–output scorer. In decentralized Proof-of-Quality, evaluator nodes assign scores $s_e(q,y)\in[0,10]$ to a query–output pair, aggregate them into a consensus estimate $\hat s(q,y)$, and use that estimate in a cost-aware reward function $\pi(i)=f(\hat s(q,y_i),c_i)$ (Tian et al., 20 Apr 2026). The same work defines multidimensional composite scoring,

$$\hat s(q,y)=\sum_{k=1}^{K}\tilde w_k z_k(q,y), \qquad \tilde w_k=\frac{w_k}{\sum_j w_j},$$

and introduces a dedicated judge dimension $z_{\text{judge}}(q,y)=s_\theta(q,y)$ produced directly from the pair $(q,y)$ without a reference answer (Tian et al., 20 Apr 2026).

A second formulation casts Q-Judger as a solution validator for verifiable reasoning. In JudgeRLVR, for a problem $x$ and a candidate solution response $z$, the judge extracts a final answer $a(z)$, defines a binary correctness label $\ell(x,z)=\mathbf{1}[a(z)=a^*(x)]$, and produces a commentary $\hat s(q,y)$0 together with a verdict token $\hat s(q,y)$1 (Duo et al., 13 Jan 2026). The judge-stage reward is itself verifiable:

$\hat s(q,y)$2

This turns judging into a discriminative precursor to generation rather than a post hoc evaluator alone (Duo et al., 13 Jan 2026).

A third formulation appears in RankJudge, where the judge must not only choose the better of two conversations but also localize the flawed turn and identify the failure category. Its strict correctness variable is

$\hat s(q,y)$3

which rejects verdict-only success and credits only judgments that recover the reason why one conversation is better (Tang et al., 20 May 2026). This makes Q-Judger a structured evaluator of conversational interaction rather than a winner-picking oracle.

2. Supervision, training objectives, and optimization

Reference-free Q-Judger training is exemplified by PoQ-Judge. It studies three judge architectures across a quality–cost tradeoff: a TextCNN judge with about 10M parameters, about 1 ms latency, and a 37 MB checkpoint; a MiniLM cross-encoder judge with 22M parameters, about 13 ms, and 87 MB; and a DeBERTa judge with 184M parameters, about 15 ms, and 702 MB (Tian et al., 20 Apr 2026). All consume a query–output pair and predict a scalar score in $\hat s(q,y)$4 through a regression head. Training proceeds in two stages: pre-train on UltraFeedback with 45k train / 5k val using mean-squared error,

$\hat s(q,y)$5

then fine-tune on GPT-labeled domain data with 1,400 train / 300 val / 300 test from QA and summarization (Tian et al., 20 Apr 2026). On the held-out test set, the best DeBERTa judge reaches 0.747 Pearson correlation with the ground-truth proxy, while the reference-free composite mode reaches 0.645 Pearson correlation, matching the best single reference-based evaluator without requiring references (Tian et al., 20 Apr 2026).

JudgeRLVR uses a two-stage judge-first, generate-second RLVR pipeline. The judge is trained first on 113k math problems with gold answers, using 16 candidate solution responses per problem sampled from MiMo-7B RL and Qwen3-30B-A3B-SFT, with hard negative mining and class balancing (Duo et al., 13 Jan 2026). The same model is then warm-started into vanilla generating RLVR. Compared to vanilla RLVR on the same math-domain data, JudgeRLVR yields an overall average accuracy increase from 76.1 to 79.8 on in-domain math while reducing overall average generation length from 23.4k to 14.8k tokens, and it improves out-of-domain average accuracy from 73.7 to 78.1 (Duo et al., 13 Jan 2026). The stated mechanism is that discriminative capability is learned before generation, so low-value search branches are pruned without any explicit length penalty (Duo et al., 13 Jan 2026).

CompassJudger-2 extends verifiable-reward training to a generalist judge model. It fine-tunes Qwen2.5-Instruct checkpoints and supervises a designated decision token with a binary reward

$\hat s(q,y)$6

combined with rejection sampling over 8 candidates per instruction and a margin loss on the decision token (Zhang et al., 12 Jul 2025). In the reported ablation, the margin variant reaches 72.11 average across judge benchmarks, compared with 69.90 for the baseline, outperforming the corresponding DPO and temperature-scaled variants (Zhang et al., 12 Jul 2025). A related invariance-oriented optimization appears in J4R, whose EIS-GRPO objective groups equivalent initial states created by swapping A/B order in pairwise judging. Starting from CompassJudger-7B, J4R-CJ-7B reaches 56.86 accuracy with 81.14 consistency on JudgeBench and 45.04 accuracy with 80.98 consistency on ReasoningJudgeBench, exceeding GPT-4o in aggregate and improving markedly over GRPO baselines that lack the equivalent-state treatment (Xu et al., 19 May 2025).

3. Benchmark design and the move beyond single-turn QA

RankJudge is the clearest explicit generalization of a Q-Judger-style single-turn QA benchmark into the setting of deployed assistants. It generates paired, multi-turn conversations grounded in the same reference document(s) and injects exactly one flaw into one turn of the worse conversation (Tang et al., 20 May 2026). The flaw taxonomy contains seven categories—self_contradiction, evasion, disorganized, fabricated_answer, instruction_forgetting, no_clarification, unnecessary_refusal—and it is crossed with seven controlled user behaviors—focused, integrative, scattered, skeptical, misinformed, exploratory, underspecified (Tang et al., 20 May 2026). A three-layer automated verification cascade checks coherence, adherence, and grounding; from 1200 generated pairs across Machine Learning, Biomedicine, and Finance, 652 survive all filtering and curation, for an overall survival 54.3% (Tang et al., 20 May 2026). Judges are then ranked with a bipartite Bradley–Terry model,

$\hat s(q,y)$7

which simultaneously estimates judge strength and per-pair difficulty (Tang et al., 20 May 2026). The benchmark reports stable rankings under partial observability, a coarser correctness criterion, and an alternative random-walk rating algorithm (Tang et al., 20 May 2026).

JudgmentBench addresses a different, but complementary, question: what supervision signal should a Q-Judger learn from in a domain without verifiable ground truth? It pairs 1,539 rubric scores and 1,530 pairwise preference judgments from 51 practicing attorneys on 30 real-world legal tasks, with the two methodologies elicited from the same experts on the same items (Yang et al., 24 May 2026). For rubrics, pooled strength is the mean expert score. For comparative judgment, pooled strength is the fitted Bradley–Terry utility,

$\hat s(q,y)$8

On the constructed three-level quality ordering, comparative judgments recover the intended ranking far better than rubrics, with mean task-level Spearman correlation 0.908 vs. 0.150, and an estimated difference 0.758 [0.494, 1.021], while requiring 1.92 minutes median per task compared with 4.74 minutes for rubrics (Yang et al., 24 May 2026). For Q-Judger design, the implication is not that rubrics are obsolete, but that pairwise preference supervision may be the more faithful primary ranking signal in tacit, high-expertise domains (Yang et al., 24 May 2026).

4. Multimodal, cohort-aware, and distributed extensions

Flex-Judge extends Q-Judger into the multimodal regime by fine-tuning only the language backbone of multimodal models on 1K curated text-only reasoning samples, while leaving modality encoders unchanged (Ko et al., 24 May 2025). It produces structured outputs in > ... and <answer>...</answer> and is instantiated as Flex-VL-7B, Flex-Omni-7B, and Flex-Mol-LLaMA for image/video, image/video/audio, and molecular modalities respectively (Ko et al., 24 May 2025). On image understanding, Flex-VL-7B reports Pearson 0.332, pairwise accuracy 0.538 with tie, 0.655 without tie, and batch distance 0.426; on GenAI-Bench, majority voting raises Flex-VL-7B from 45.17 to 49.29 overall, surpassing the cited GPT-4o overall number of 49.20 (Ko et al., 24 May 2025). In molecular evaluation, judge-guided best-of-$\hat s(q,y)$9 improves Mol-LLaMA accuracy on PAMPA from 72.48% base to 77.49% with $\pi(i)=f(\hat s(q,y_i),c_i)$0, and DPO using 4,253 judge-labeled preferences pushes accuracy to 80.10% (Ko et al., 24 May 2025). This supports a multimodal conception of Q-Judger as a transferable reasoning-based evaluator rather than a text-only grader.

M-JudgeBench and M-Judger reformulate multimodal judging around capability axes rather than task names. M-JudgeBench contains 3,712 instances divided into 1,364 pairwise CoT, 1,610 length-bias, and 738 process-error examples, spanning ten fine-grained subtasks (Chen et al., 28 Feb 2026). Its data construction relies on Judge-MCTS, which generates Short-Correct, Short-Error, Long-Correct, Long-Error trajectories so that judges can be trained explicitly against length bias and process-error blindness (Chen et al., 28 Feb 2026). On M-JudgeBench, Qwen3-VL-8B-Instruct rises from 50.78% base to 57.46% with SFT plus MCTS data and to 62.42% with the full RL configuration, while length-bias accuracy increases from 24.66% to 44.53% (Chen et al., 28 Feb 2026). The result is a capability-oriented multimodal Q-Judger that is evaluated on reasoning style, response length, and subtle process defects rather than only answer correctness (Chen et al., 28 Feb 2026).

A further generalization is Judge Agent Forest, which makes judging cohort-aware. Instead of evaluating each query–response pair in isolation, the judge evaluates a focal pair together with a sampled set of peer exemplars from the same cohort, and repeated randomized judging defines a robustness score

$\pi(i)=f(\hat s(q,y_i),c_i)$1

In the reported cloud misconfiguration triage study over 315 cloud assets, even a naive JAF with simple peer sampling shifts mass toward high correctness probabilities and converges faster than an isolated judge after 5 and 10 refinement iterations (Garg et al., 29 Jan 2026). This suggests that Q-Judger can also function as a collective, relation-aware evaluator rather than only a local comparator (Garg et al., 29 Jan 2026).

5. Reliability, bias, and measurement

A major line of work argues that Q-Judger should be treated as a measurement instrument, not as a generic scalar oracle. “Evaluative Fingerprints” reports 3,240 evaluations from 9 judges over 120 video–pack items and finds Krippendorff’s $\pi(i)=f(\hat s(q,y_i),c_i)$2 overall, with negative $\pi(i)=f(\hat s(q,y_i),c_i)$3 on Readability & Structure and SEO Mechanics (Nasser, 8 Jan 2026). At the same time, judges are stable with themselves: Gemini-3-Pro reaches ICC(3,1) = 0.872, GPT-5.2 0.845, and Claude-Opus 0.811 (Nasser, 8 Jan 2026). A classifier can identify which judge produced an evaluation with 77.1% accuracy from rubric scores alone and 89.9% with disposition features, while GPT-4.1 and GPT-5.2 are separable at 99.6% within-family (Nasser, 8 Jan 2026). The paper’s conclusion is not that judges are noisy, but that they encode different stable theories of quality; averaging them produces a synthetic verdict that corresponds to no judge’s actual values (Nasser, 8 Jan 2026).

“The Judge Who Never Admits” studies hidden shortcut use by injecting synthetic metadata cues and measuring both Verdict Shift Rate (VSR) and Cue Acknowledgment Rate (CAR). Under recency cues on ELI5, for example, GPT-4o shows 30 / 0, Qwen3-235B 32 / 57, and Claude-3-Haiku 37 / 0 for VSR% / CAR%; on LitBench, Claude-3-Haiku reaches 71 / 0 for recency and 65 / 1.5 for educational status (Marioriyad et al., 8 Feb 2026). The paper emphasizes the resulting explanation gap: substantial verdict sensitivity can coexist with near-zero explicit acknowledgment of the cue, especially in open-ended creative evaluation (Marioriyad et al., 8 Feb 2026). A common misconception is therefore that the rationale reliably reveals the factors driving the verdict; the reported CAR values show that this is often false (Marioriyad et al., 8 Feb 2026).

“Judge Circuits” adds a mechanistic diagnosis. Using Position-aware Edge Attribution Patching (PEAP), it identifies a sparse, shared Latent Evaluator subgraph in mid-to-late MLPs and some late heads, with format-specific Task Formatters that map a stable latent signal to output tokens (Feldhus et al., 15 May 2026). Across tasks, 21/25 model–task cells achieve median recovery $\pi(i)=f(\hat s(q,y_i),c_i)$4 with $\pi(i)=f(\hat s(q,y_i),c_i)$5 edges, and Gemma-3-27B RewardBench saturates near 1.0 with $\pi(i)=f(\hat s(q,y_i),c_i)$6 edges (Feldhus et al., 15 May 2026). Zero-ablation of the Latent Evaluator collapses judgment while preserving world knowledge in architecturally modular models such as Qwen2.5-14B, where Clinical DB remains 87% → 87% and StrategyQA 72.0% → 72.0% (Feldhus et al., 15 May 2026). The paper’s claim is that cross-format inconsistency is partly a problem of formatter geometry, not of the underlying judgment signal itself (Feldhus et al., 15 May 2026).

The psychometric extension of this view appears in “LLM Judges Have Dark Current,” which proposes a Judge Datasheet measuring dark current, stable cross-sensitivity, positional false preference, target sensitivity, and the criterion induced by tie instructions (Usami et al., 14 Jun 2026). In the reported case study, Qwen2.5-32B has DC=0.0000, Raw $\pi(i)=f(\hat s(q,y_i),c_i)$7 FP=0.2583, SCS=0.0000, PFP=0.0833, $\pi(i)=f(\hat s(q,y_i),c_i)$8, and $\pi(i)=f(\hat s(q,y_i),c_i)$9 (Usami et al., 14 Jun 2026). When a strict tie criterion is applied, its $$\hat s(q,y)=\sum_{k=1}^{K}\tilde w_k z_k(q,y), \qquad \tilde w_k=\frac{w_k}{\sum_j w_j},$$0 raw false preference drops to 0.0000, but $$\hat s(q,y)=\sum_{k=1}^{K}\tilde w_k z_k(q,y), \qquad \tilde w_k=\frac{w_k}{\sum_j w_j},$$1 target sensitivity falls from 0.9400 to 0.5000, while $$\hat s(q,y)=\sum_{k=1}^{K}\tilde w_k z_k(q,y), \qquad \tilde w_k=\frac{w_k}{\sum_j w_j},$$2 remains 1.0000 (Usami et al., 14 Jun 2026). The stated lesson is that prompting moves the criterion, not the resolution (Usami et al., 14 Jun 2026).

6. Applications, limits, and research directions

Q-Judger is already positioned as infrastructure for training data selection, checkpoint ranking, and release gating in interactive LLM systems, precisely because humans cannot feasibly evaluate every candidate conversation (Tang et al., 20 May 2026). In decentralized inference, it serves as the reference-free semantic component of Proof-of-Quality reward allocation (Tian et al., 20 Apr 2026). In reasoning models, a judge-first stage can be used to initialize a generator that later solves problems more directly and with less aimless exploration (Duo et al., 13 Jan 2026). In multimodal and scientific settings, the same general idea drives best-of-$$\hat s(q,y)=\sum_{k=1}^{K}\tilde w_k z_k(q,y), \qquad \tilde w_k=\frac{w_k}{\sum_j w_j},$$3 selection, reranking, and DPO data curation in areas such as image generation, audio MOS, and molecular property prediction (Ko et al., 24 May 2025).

The literature also converges on several limitations. RankJudge notes that its coverage is restricted to Machine Learning, Biomedicine, and Finance, that its conversations are synthetic, that the benchmark uses static reference documents, and that the “disorganized” flaw is particularly challenging to stage (Tang et al., 20 May 2026). PoQ-Judge identifies proxy quality as the main remaining limitation, since results are much stronger on QA than on summarization under a token-level F1 proxy (Tian et al., 20 Apr 2026). Flex-Judge reports that performance remains modality-dependent, with batch-level ranking on image understanding and task-specific audio evaluation still challenging, and that too much text-only fine-tuning can cause catastrophic forgetting (Ko et al., 24 May 2025). M-JudgeBench shows that smaller models remain weak at process-error detection, especially for incidental mistakes where two responses are nearly textually identical (Chen et al., 28 Feb 2026).

A plausible implication is that no single Q-Judger protocol is sufficient across all evaluation regimes. Single-turn query–output regression, verifiable binary verdicts, pairwise comparative judgment, multimodal reasoning-first supervision, cohort-aware context sharing, and psychometric datasheets address different parts of the judging problem rather than interchangeable variants of one solved method. The current research direction therefore treats Q-Judger as a family of evaluative instruments whose design must be matched to the target construct, whose supervision signal must be justified, and whose failure modes—format dependence, hidden shortcut use, positional bias, criterion drift, and inter-judge disposition—must be measured explicitly rather than assumed away.