Topic Judge: Legal & AI Evaluation

• Topic Judge is a dual concept in legal NLP and AI evaluation, representing both human judicial decision-making and automated outcome scoring.
• In legal NLP, structured datasets analyze judges' discourse to predict outcomes and support tasks like judgment retrieval and summarization.
• In AI evaluation, judge models employ reasoning-intensive methods to assess model outputs across multimodal benchmarks, addressing bias and consistency.

In recent computational research, the term judge denotes two closely related but technically distinct entities: the human judicial decision-maker modeled in legal NLP, and the automated evaluator—typically an LLM or MLLM—used to score, rank, or critique model outputs. In the legal setting, judges appear as authors of structured opinions, as carriers of institution-specific discourse conventions, and, in some studies, as individually predictive variables in outcome modeling. In AI evaluation, judge models operationalize preference comparison, rubric-based scoring, critique generation, and multimodal assessment across text, image, audio, and video. Across both literatures, the judge is increasingly treated not as an interchangeable oracle but as a structured source of reasoning, heterogeneity, and potential bias (Xuan et al., 4 Jun 2026, Chen et al., 31 Mar 2025).

1. Judges and judgments in legal NLP

Legal NLP treats judicial output as highly structured discourse rather than unorganized prose. The Hong Kong Judgment Discourse Dataset (HKJudge) is described as the first sentence-level expert-annotated legal discourse corpus for Hong Kong judgments and covers criminal judgments from all five levels of the Hong Kong court hierarchy. It contains approximately 290k sentences, approximately 6.5 million tokens, 4,000 documents, 292,240 sentence-tag pairs, and 26 sentence-level rhetorical roles grouped under Fact, Inference, Result, and Other; Result sentences are further annotated with three span-level sentencing elements: charge, imprisonment term, and fine. The annotation was produced by 10 legal linguistics annotators with inter-annotator agreement $\kappa = 0.8$, and the corpus is used to define rhetorical role classification and legal element extraction tasks (Xuan et al., 4 Jun 2026).

A second line of work addresses judgment retrieval and summarization. “Court Judgement Labeling Using Topic Modeling and Syntactic Parsing” builds a heuristic system for Hong Kong judgments that begins with Aspect-driven Topic Modeling for sentence selection and then uses dependency parsing, constituency parsing, sentence simplification, and a legal term tree for phrase-level tag generation. The Hong Kong legal concept tree contains more than 13,000 common legal terms adapted from Sweet & Maxwell’s taxonomy. In the reported experiment on 15 independent Hong Kong personal-injury judgments evaluated by 7 participants, the hybrid method yielded 6.80 tags, 1.76 good tags, and a 0.258 good-tag rate, while word tags yielded 10.60 tags, 1.58 good tags, and a 0.149 rate; the paper interprets the hybrid method as preferable for labeling quality, even though phrase tags were not yet clearly superior for recommendation (Liu, 2022).

Legal generation work extends the same concern for structure from labeling to full drafting. JuDGE defines judgment document generation for the Chinese legal system as generating a complete criminal judgment from a factual description and provides 2,505 fact–judgment pairs, together with a statute/regulation corpus containing 55,348 statutory articles and a prior-judgment corpus containing 103,251 judgment documents. Its evaluation framework, developed with legal professionals, measures penalty accuracy, convicting accuracy, referencing accuracy, and documenting similarity, and the paper reports that multi-source retrieval-augmented generation is the strongest baseline while still leaving substantial room for improvement (Su et al., 18 Mar 2025).

2. The judge as a predictive variable

Where discourse-oriented legal NLP focuses on how judges write, another line of work asks whether who the judge is measurably affects outcomes. “The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction” studies French appellate child physical custody rulings and frames the problem against the legal realism–formalism debate. The final dataset contains 18,937 rulings extracted from 10,306 cases, with class distribution 67.3% mother custody, 22.3% father custody, and 10.5% shared custody. The pipeline combines Llama 3.3 70B for structured feature extraction with RF, XGB, and SVC for prediction, and compares specialist models trained on one judge’s rulings with a pooled judge-agnostic baseline. The top specialist result reaches 92.85% F1, while the best reported generalist model reaches 82.63% despite using 20x to 100x more samples; specialist models beat the generalist 13/13 times for RF and 9/13 times for both XGB and SVC in the reported summary (Zambrano, 18 Jul 2025).

The paper interprets this pattern as evidence that specialist models learn stable individual decisional regularities that do not transfer well across judges. Its In-Domain and Cross-Domain tests are designed precisely to separate learnability from transferability, and the reported drop under cross-judge transfer is treated as empirical support for legal realism rather than a judge-neutral view of adjudication (Zambrano, 18 Jul 2025).

This judicial-identity perspective complements discourse corpora such as HKJudge. HKJudge formalizes judgments as a progression from fact-finding to reasoning to ruling, whereas judge-variable studies ask whether the mapping from structured facts to outcomes is invariant across decision-makers. Taken together, these works model the judge simultaneously as a discourse producer and as an outcome-bearing latent variable (Xuan et al., 4 Jun 2026).

3. LLM-as-a-judge and reasoning-centric evaluation

In LLM evaluation, a judge is a model that compares candidate outputs, assigns scores, or produces critiques. A central claim in this literature is that judging is reasoning-intensive rather than a shallow classification problem. JudgeLRM explicitly argues that judgment requires evidence verification, error detection, comparison across multiple criteria, calibration, and justification. The paper reports a negative linear relationship between SFT improvement and reasoning requirement on PandaLM categories, with fitted line $y = -0.41x + 16.72$ and $R^2 = 0.53$, and trains reasoning-oriented judge models with GRPO and judge-specific outcome rewards. On PandaLM, JudgeLRM-7B reaches 75.05 F1 versus 67.98 for the corresponding SFT baseline; the paper also reports that JudgeLRM-8B reaches 75.26 F1 and JudgeLRM-14B reaches 76.29 (Chen et al., 31 Mar 2025).

CompassJudger-2 pursues a different generalization strategy: a task-driven, multi-domain curation pipeline supervised with verifiable rewards and refined with margin policy gradient loss. The paper distinguishes Public Judge Data, Public Reward Data transformed via rejection sampling, synthetic knowledge/chat judgments, and general instruction data, and evaluates on RewardBench, JudgeBench, RMB, and the newly introduced JudgerBenchV2. CompassJudger-2-7B reports 60.52 on JudgerBenchV2, 63.06 on JudgeBench, 73.90 on RMB, and 90.96 on RewardBench, for an average of 72.11; the 32B model reports an average of 73.32. The paper attributes part of the gain to verifiable rewards and part to margin-based optimization, and also emphasizes improved prompt robustness (Zhang et al., 12 Jul 2025).

For retrieval-augmented generation, ConsJudge addresses prompt sensitivity directly. It constructs multiple hybrid evaluation aspects from four dimensions—hallucination, completeness, coherence, and semantic consistency—generates multiple judgments, and computes a judge-consistency score

$$S_i = \frac{1}{k}\sum_{j=1}^{k}\cos(\mathrm{Emb}(r_i),\mathrm{Emb}(r_j)).$$

The most consistent judgment becomes the positive instance and the least consistent judgment the negative instance for DPO training. The paper reports that ConsJudge yields higher agreement with GLM-4-plus and humans than raw metrics or vanilla prompting, and that removing the consistency mechanism degrades performance (Liu et al., 26 Feb 2025).

4. Multimodal judges and benchmark design

Multimodal judging extends the evaluator beyond text to any-to-any modality mappings. JudgeAnything introduces TaskAnything and JudgeAnything as unified benchmarks for multimodal understanding and generation across 15 any-to-any modality categories. TaskAnything contains 1,500 queries with 100 queries per task, while JudgeAnything contains 9,000 judging instances and evaluates judges under Pair Comparison and Score Evaluation. The reported summary result is that MLLM judges perform much better on multimodal understanding than multimodal generation: 66.55% in Pair Comparison and 42.79% in Score Evaluation for MMU, versus 53.37% and 30.05% for MMG. The paper also states that output modality matters more than input modality, and that text/image outputs are judged more reliably than video/audio outputs (Pu et al., 21 Mar 2025).

M-JudgeBench reframes the benchmark problem from task coverage to capability coverage. It contains 3,712 instances divided into ten fine-grained subtasks spanning pairwise CoT comparison, length bias avoidance, and process error detection. The benchmark construction uses seed datasets including MMMU, MMMU-Pro, MMStar, MMReason, M3CoT, MathVision, and MathVerse, and the accompanying Judge-MCTS framework generates short/long and correct/error trajectories for training. In the reported results, M-Judger-RL-Qwen8B reaches 80.75 on VL-RewardBench, 65.52 on Multimodal RewardBench, and 62.42 overall on M-JudgeBench, which the paper presents as the best reported performance across the three judge benchmarks (Chen et al., 28 Feb 2026).

Several papers replace direct scalar scoring with explicit reasoning or rubric-conditioned outputs. MR. Judge reformulates multimodal judgment as a reasoning-based multiple-choice problem over candidate responses, with a > trace and a discrete final label. Its training uses reverse response candidate synthesis and text-based reasoning extraction, and MR. Judge-7B-SFT-RL reports 75.5 overall accuracy and 71.1 macro accuracy on VL-RewardBench; with 10-shot majority voting, the paper reports 79.5 overall (Pi et al., 19 May 2025). Flex-Judge instead asks whether a small amount of text-only reasoning supervision can transfer across modalities. Built from approximately 1K curated reasoning examples, it fine-tunes Qwen2.5-VL-7B and Qwen2.5-Omni-7B backbones and evaluates across images, video, audio, and molecules, arguing that structured textual reasoning encodes decision patterns that generalize beyond text (Ko et al., 24 May 2025).

A more compact but explicitly reproducible evaluation design appears in “Judge Model for Large-scale Multimodality Benchmarks”. That paper defines a 280-sample benchmark across text, audio, image, and video, uses Gemini-3-pro as the judge, and requires the judge to produce a Likert-style score from 0 to 5 together with an error type and explanation. The benchmark aggregates mean scores from three human annotators and the judge, and the paper reports that the judge is slightly more conservative, typically scoring about 0.1–0.3 points lower than average human scores while preserving relative model ranking nearly perfectly (Shih et al., 3 Jan 2026).

5. Judge heterogeneity, aggregation, and rank inference

A major recent shift is from judge-agnostic aggregation to explicitly judge-aware inference. One formulation extends Bradley–Terry–Luce with judge-specific discrimination:

$$\mathbb{P}(Y=1\mid i,j,k)=\sigma(\gamma_k(s_i-s_j)),$$

where $s_i$ and $s_j$ are latent item qualities and $\gamma_k$ is judge $k$’s discrimination parameter. “A Judge-Aware Ranking Framework for Evaluating LLMs without Ground Truth” proves identifiability up to the usual location/scale normalizations, establishes consistency and asymptotic normality of the MLE, and reports higher agreement with human rankings than unweighted BTL. On Chatbot Arena, the weighted ranking achieves Spearman correlation $\rho = 0.9955$ with human evaluation rankings, versus $y = -0.41x + 16.72$0 for the unweighted ranking (Xu et al., 29 Jan 2026).

“Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence” goes further by decomposing the judge–item score matrix as

$y = -0.41x + 16.72$1

Here $y = -0.41x + 16.72$2 is the consensus ranking, $y = -0.41x + 16.72$3 is judge-specific sensitivity to that consensus, and $y = -0.41x + 16.72$4 is a structured residual disagreement term. The paper develops identifiability conditions, an anchored proximal alternating algorithm, fixed-panel asymptotics, and confidence intervals for contrasts, sensitivities, and pairwise probabilities. Its empirical claim is that HJA improves recovery, robustness, uncertainty calibration, and near-tie performance relative to pooled BTL and sensitivity-only baselines (Yu et al., 6 May 2026).

Dynamic aggregation appears in LLM Jury-on-Demand. Rather than using a fixed panel, the framework trains judge-specific reliability predictors from structural, lexical, and embedding-based features and then selects the top-$y = -0.41x + 16.72$5 judges per instance. Aggregation uses normalized reliability weights

$y = -0.41x + 16.72$6

Across summarization and RAG tasks, the paper reports that Jury-on-Demand consistently outperforms all single-judge and static-jury baselines in Kendall’s Tau. For RAG groundedness, for example, Jury-on-Demand reaches $y = -0.41x + 16.72$7, compared with $y = -0.41x + 16.72$8 for Static Average-All, $y = -0.41x + 16.72$9 for Static Weighted-Regression, and $R^2 = 0.53$0 for the best single judge, GPT-OSS-120B (Li et al., 1 Dec 2025).

This body of work treats judge heterogeneity as a first-class inferential target rather than as noise. In legal prediction, that heterogeneity appears as judge-specific decisional style; in LLM evaluation, it appears as discrimination, sensitivity, residual disagreement, or context-dependent reliability (Zambrano, 18 Jul 2025, Yu et al., 6 May 2026).

6. Bias, faithfulness, and operational stability

Judge quality is not exhausted by accuracy or benchmark agreement. “The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation” studies six cue families—source, temporal, age, gender, ethnicity, and educational status—using controlled metadata perturbations on ELI5 and LitBench. The paper introduces Verdict Shift Rate (VSR) and Cue Acknowledgment Rate (CAR), and reports large behavioral effects for provenance hierarchies, recency preferences, and educational-status favoritism while CAR remains typically at or near zero. The reported preference ordering for provenance is Expert $R^2 = 0.53$1 Human $R^2 = 0.53$2 LLM $R^2 = 0.53$3 Unknown, and the central finding is the combination of substantial verdict sensitivity with limited explicit cue acknowledgment (Marioriyad et al., 8 Feb 2026).

Operational reliability poses a different problem: the judge itself can drift. “Who Drifted: the System or the Judge?” addresses the ambiguity between system degradation and evaluator change by interleaving a fixed human-labeled anchor set with production monitoring and running a second anytime-valid e-process on the judge–human gap. Its guard-window rule returns one of three verdicts—none, system, or judge. In experiments on two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300; the paper also reports that the industry-default rolling z-test false-alarms on 75% of drift-free streams (Li, 13 Jun 2026).

Prompt sensitivity appears repeatedly as a failure mode. ConsJudge treats inconsistency across evaluation aspects as a training signal (Liu et al., 26 Feb 2025); CompassJudger-2 explicitly evaluates prompt robustness (Zhang et al., 12 Jul 2025); JudgeAnything reports that fine-grained rubrics help Score Evaluation more consistently than Pair Comparison, but can also induce hallucinations or ties when criteria are mismatched to the task (Pu et al., 21 Mar 2025). These results indicate that a judge can be simultaneously accurate in aggregate and brittle with respect to cueing, prompting, or API-level change.

7. Domain-specific judge constructions and broader applications

Not all judge systems are generic evaluators. In information retrieval, “Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs” proposes one monoT5 + LoRA adapter per topic and assessor, trained to extend incomplete relevance pools. The paper reports that system rankings induced by these classifiers achieve Spearman’s $R^2 = 0.53$4 with ground-truth system rankings, and that as little as 128 initial human judgments per topic suffice to improve comparability relative to treating unjudged documents as non-relevant (Gienapp et al., 6 Oct 2025).

In topic modeling, “Model Directions, Not Words” introduces topic judge, an LLM-based pairwise framework that compares topic sets conditioned on a specific document rather than judging topic coherence in isolation. The method samples a document, extracts one or two top topics per model, asks the judge to choose which topic set better captures the document’s meaning, and aggregates pairwise wins through a Bradley–Terry model and Elo ratings. The reported evaluation covers five datasets with 100 comparisons per model pair per dataset, for a total of 2800 comparisons (Zheng et al., 31 Jul 2025).

In agentic systems, JAF reconceives the judge as a cohort-level inferential component rather than a per-instance scorer. For each focal query–response pair, the judge is prompted with a neighborhood of related peers and outputs an accept/refine label plus critique. The paper formalizes repeated randomized judging through empirical acceptance probabilities

$R^2 = 0.53$5

and reports that, in cloud misconfiguration triage, JAF improves correctness probability, robustness, and efficiency relative to isolated judge self-refinement (Garg et al., 29 Jan 2026).

These specialized constructions suggest that the judge is increasingly treated as a configurable systems component: sometimes a discourse annotator, sometimes a legal outcome model, sometimes a multimodal benchmark instrument, and sometimes a reliability-weighted or cohort-aware inference layer. The common research direction is not toward a single universal judge, but toward explicit modeling of what is being judged, under which protocol, by which evaluator, and with what uncertainty (Xu et al., 29 Jan 2026).