---
title: 'SelfJudge: Unsupervised LLM Self-Evolution'
url: https://www.emergentmind.com/topics/selfjudge
type: topic
---

# SelfJudge: Unsupervised LLM Self-Evolution

“SelfJudge” denotes two related but non-equivalent ideas in contemporary LLM evaluation. In the narrow sense, it refers to the unsupervised multimodal self-evolution framework in which one model instantiates an **Actor** and a frozen **Judge**, samples multiple reasoning trajectories on unlabeled image–question inputs, and improves itself using self-consistency, bounded judge modulation, and group-relative policy optimization [2603.21289]. In a broader sense, the term overlaps with the wider design space of **LLM-as-a-judge** systems, including same-family evaluators, self-improving judges, and externalized judge models used to score or compare model outputs [2410.09775][2502.18817]. The distinction is consequential: some systems study a model’s own evaluation of its generations, whereas others study standalone evaluator models whose relevance to “self-judging” is indirect but substantial.

## 1. Conceptual scope

In strict usage, SelfJudge is not merely “an LLM used as a judge,” but a training architecture in which the model improves on unlabeled data through internally generated supervisory signals [2603.21289]. The broader literature, however, often uses closely related language for any judge model that evaluates model outputs, including rubric-conditioned open judges, reward-model-like evaluators, or role-split judge agents. This broader usage is technically useful but conceptually looser.

That distinction is explicit in the surrounding literature. EasyJudge, for example, is best understood as an **open-source LLM-as-a-judge system for evaluating LLM outputs**, not as a strict self-judging method; it evaluates responses from other models or arbitrary candidate responses rather than a model explicitly critiquing its own output [2410.09775]. By contrast, JAF uses the same underlying LLM family for both the **primary agent** and the **judge agent**, with different prompts implementing the distinct roles, which places it closer to role-prompted self-judging and iterative self-refinement [2601.22269].

The broader SelfJudge landscape is therefore heterogeneous. It includes: a narrow self-evolution framework over unlabeled multimodal data; cohort-aware self-refinement systems; cross-prompt self-improving judges; post-hoc calibrated judges; and externalized evaluator platforms. A technically precise account must keep these senses separate.

## 2. SelfJudge as unsupervised self-evolution

In the narrow formulation, SelfJudge addresses multimodal reasoning post-training without human-annotated answers, external reward models, verifier labels, or teacher-model distillation [2603.21289]. For each unlabeled input \(x=(I,q)\), the Actor policy \(\pi_\theta\) samples a group of reasoning trajectories,
\[
\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),
\]
and each trajectory \(\tau_i\) ends in a final answer \(a_i\).

The first training signal is the Actor’s own self-consistency distribution. If \(\hat p(a)\) is the empirical frequency of answer \(a\) in the sampled group, then each trajectory receives the soft prior
\[
r_i^{\mathrm{SC}} = \hat p(a_i).
\]
This departs from majority-vote pseudo-labeling, which would collapse the group to a single answer. The paper’s claim is that soft self-consistency preserves more within-group structure than hard majority reward and avoids amplifying an early dominant mode as aggressively [2603.21289].

A second role is played by the Judge. The Judge is not an external model; it is initialized as a structurally identical copy of the current Actor at the start of training and then frozen. Given input \(x\) and trajectory \(\tau_k\), it outputs a score
\[
s_k = J_\phi(x, \tau_k), \quad s_k \in [0,1].
\]
According to the appendix prompt, this score aggregates **answer correctness**, **reasoning quality**, and **visual grounding**, with weights \(0.50\), \(0.30\), and \(0.20\), respectively; a mandatory validity rule assigns all-zero scores if the response lacks a clear single final answer in the required format [2603.21289].

This architecture makes SelfJudge neither pure self-consistency training nor conventional reward-model RL. It is a closed-loop system in which the Actor supplies a low-variance prior from answer agreement, while the frozen Judge supplies a quality-sensitive correction signal.

## 3. Reward shaping, distributional modeling, and GRPO

SelfJudge does not use the raw Judge score directly as reward. Instead, the Judge modulates the self-consistency prior through a bounded function
\[
g(s) = 1 + \lambda_{+}\,\sigma\!\Big(\frac{s - t_h}{\tau_h}\Big) - \lambda_{-}\,\sigma\!\Big(\frac{t_l - s}{\tau_l}\Big),
\]
chosen to be continuous, bounded, and centered around \(1\) so that high-quality trajectories are amplified and low-quality trajectories are suppressed without letting the Judge completely override the Actor prior [2603.21289]. The final reward is
\[
R_k = r_k \cdot g(s_k) - \lambda_{\mathrm{fmt}}\, \delta_k,
\]
where \(\delta_k\) marks output-format violations and \(\lambda_{\mathrm{fmt}} = 0.5\).

The method then converts absolute rewards into a within-group distribution. With \(\tilde r_k = \alpha R_k\),
\[
q_\alpha(\tau_k\mid x) = \frac{\exp(\alpha R_k)}{\sum_{j=1}^{n}\exp(\alpha R_j)},
\qquad
A_k(x) = \log q_\alpha(\tau_k\mid x).
\]
This group-relative construction is central. The paper argues that absolute Judge scores are not comparable across inputs, whereas relative advantages inside the sampled trajectory set are more stable and reduce collapse [2603.21289].

Optimization uses **Group Relative Policy Optimization (GRPO)** with clipping and KL regularization. In the main experiments, both Actor and Judge start from **Qwen2.5-VL-7B-Instruct**; training uses **8 trajectories** per question, **20 epochs**, **AdamW**, one node with \(8\times\) NVIDIA A800 80GB GPUs, learning rate \(1\times 10^{-6}\), KL coefficient \(\beta=0.01\), \(\alpha=1\), and reward-modulation hyperparameters
\[
\lambda_+=\lambda_-=0.2,\quad t_h=0.95,\quad t_l=0.40,\quad \tau_h=\tau_l=1
\]
[2603.21289]. The Actor prompt enforces reasoning inside `<think>...</think>` and a single final answer in `\boxed{ANSWER}`.

The unlabeled training data come from **Geometry3K (Geo3K)**, **GeoQA**, and **MMR1**, with labels discarded. Evaluation is reported on **MathVision**, **MathVerse**, **WeMath**, **LogicVista**, and **DynaMath**, with additional checks on **ChartQA** and **MMVP** [2603.21289].

## 4. Related self-judging and judge-improvement paradigms

The narrow SelfJudge framework sits inside a broader family of judge architectures. JAF turns self-judging into a **cohort-level** process: a judge evaluates each query–response pair together with a small neighborhood of peer examples, inducing a graph over the cohort and enabling critique propagation across related instances [2601.22269]. In the reported experiments, both the primary agent and the judge use **Llama3.3-Nemotron-Super-49B-v1.5** with different prompts, and repeated randomized judging yields an empirical acceptance probability
\[
\hat{p}_i = \frac{1}{R}\sum_{r=1}^R \mathbf{1}\{J_i^{(r)}=\text{accept}\},
\]
which functions as a stability signal for self-evaluation [2601.22269].

ConsJudge addresses a different failure mode: prompt-sensitive judge instability in **RAG** evaluation. It generates judgments under \(k=8\) hybrid aspect prompts built from **hallucination**, **completeness**, **coherence**, and **semantic consistency**, embeds those judgments, and chooses the most consensus-like and least consensus-like outputs as preference pairs for **DPO** training [2502.18817]. This is not strict self-judging either, but it is a direct method for improving a judge using its own cross-prompt agreement structure.

JudgeLRM pushes the same direction from the optimization side. It treats judging as a reasoning-intensive task and trains judge models with **GRPO** and outcome-driven rewards over structured `<think>` and `<answer>` outputs [2504.00050]. On PandaLM, **JudgeLRM-7B**, **JudgeLRM-8B**, and **JudgeLRM-14B** surpass **DeepSeek-R1** by more than **2%** in F1, while **JudgeLRM-3B** and **JudgeLRM-4B** exceed **GPT-4** on the same benchmark [2504.00050]. The implication for SelfJudge is that evaluator quality depends strongly on reasoning-oriented optimization, not only on evaluator prompting.

Flex-Judge generalizes the judge-building problem across modalities. It shows that **1K** text-only reasoning examples can train a multimodal judge that transfers to image, video, audio, and molecule evaluation, and that the resulting judge can support **best-of-\(N\)** selection and **DPO** preference generation in the molecular setting [2505.18601]. This is closely aligned with SelfJudge as a practical recipe for self-reward or same-family judge construction.

By contrast, EasyJudge represents the externalized end of the design space: an **open-source**, **pairwise/pointwise**, rubric-conditioned evaluator with a **Streamlit-based web GUI**, visualization, and auxiliary reference-based metrics, but not a method in which a model evaluates itself as part of an intrinsic training loop [2410.09775].

## 5. Reliability, bias, and judge evaluation methodology

A major result of the wider literature is that judge reliability cannot be assumed, even when the judge is strong. JudgeSense formalizes prompt sensitivity through the **Judge Sensitivity Score (JSS)**,
\[
\mathrm{JSS}(j,t)=\frac{1}{|P|}\sum_{i=1}^{|P|}\delta(j(p_i),j(p_i')),
\]
the fraction of paraphrase pairs on which a judge returns the same decision [2604.23478]. On coherence, JSS ranges from **0.389** to **0.992** across models. On factuality, judges cluster near **0.63** in raw form because of a polarity-inverted prompt artifact; after correction, factuality JSS rises to about **0.9** or higher. In pairwise preference and relevance, **8 of 9 judges** collapse into always-A behavior, making naive perfect consistency largely meaningless [2604.23478].

CyclicJudge studies a different pathology: persistent judge-specific bias. Its variance decomposition,
\[
Var(\bar{X}) = \frac{\sigma_\alpha^2}{n} + \frac{\sigma_\beta^2}{nm} + \frac{\sigma_\varepsilon^2}{nmK} + \frac{\sigma_\gamma^2}{K}\cdot\frac{J-K}{J-1},
\]
shows that judge bias does not vanish by increasing the number of scenarios or sampled generations; it vanishes only by increasing the number of distinct judges or balancing them by design [2603.01865]. On MT-Bench, the paper reports concrete self-preference effects: under **Qwen-as-judge**, Qwen ranks itself first even though the paper says it is actually the weakest of the three evaluated models by the consensus of the other judges [2603.01865]. For SelfJudge, this is direct evidence that self-evaluation can materially reverse rankings.

Metric choice is also part of judge methodology. For prevalence-estimation tasks such as pass rate, refusal rate, or policy-violation rate, the paper on **Balanced Accuracy** argues that the right criterion for choosing a judge is **Youden’s \(J\)**, or equivalently **Balanced Accuracy**, because judged prevalence obeys
\[
y = \mathrm{TPR}\cdot x + \mathrm{FPR}\cdot(1-x),
\]
so the preservation of true model differences depends on \(J=\mathrm{TPR}-\mathrm{FPR}\), not on raw Accuracy or F1 [2512.08121]. In parallel, **LLM Jury-on-Demand** treats reliability as instance-dependent and learns per-judge reliability predictors; on RAG groundedness it reaches **0.68 ± 0.02** Kendall’s Tau versus **0.58 ± 0.02** for static Average-All, and on RAG completeness **0.50 ± 0.03** versus **0.38 ± 0.03** [2512.01786].

FairJudge integrates many of these concerns into training itself. It models judging as a policy
\[
\pi_{\text{judge}}(y \mid x, c, m),
\]
constructs **FairJudge-16K** plus **FairJudge-Benchmark-1K**, and uses a curriculum **SFT-DPO-GRPO** pipeline to target rubric adherence, debiasing, and cross-mode consistency [2602.06625]. This line of work suggests that SelfJudge is increasingly less about a single judge prompt and more about explicit control over judging behavior.

## 6. Empirical standing, applications, and limits

On its own core task, SelfJudge reports consistent gains over majority-vote self-training. With **Geo3K** as unlabeled training data, average accuracy across the five mathematical reasoning benchmarks rises from **34.6** to **37.9**, and **MathVision** rises from **25.0** to **30.9** [2603.21289]. Compared with **MM-UPT**, SelfJudge is consistently stronger: with Geo3K training, MM-UPT averages **35.4** while SelfJudge reaches **37.9**; on MathVision the corresponding numbers are **27.5** and **30.9**, and on DynaMath **21.4** and **24.2** [2603.21289].

The ablations isolate the method’s ingredients. Starting from **Qwen2.5-VL-7B**, the paper reports on MathVision / DynaMath: - base **25.0 / 20.3**; - **+ Major Vote**: **27.5 / 21.4**; - **+ Self-Consistency**: **25.2 / 20.5**; - **+ Judge Scoring**: **27.3 / 21.1**; - **+ MV + JS**: **28.4 / 22.7**; - **+ SC + JS**: **30.1 / 23.7**; - **+ SC + JS (Dist.)**: **30.9 / 24.2** [2603.21289]. The interpretation is straightforward: self-consistency alone is weak, judge scoring alone is better but unstable, and the combination with distributional modeling yields the strongest results.

The framework also shows some broader generalization. Under **MMR1** training, the base model scores **85.7 / 77.2** on **ChartQA / MMVP**, MM-UPT gives **85.4 / 74.6**, and SelfJudge gives **87.2 / 79.8** [2603.21289]. At the same time, the paper explicitly identifies a trade-off in sample diversity: on **MathVision / DynaMath**, pass@10 drops from **0.66 / 0.48** in the base model to **0.64 / 0.43** under SelfJudge, though this is still stronger than majority vote and slightly better than supervised GRPO on average [2603.21289].

The sharpest limitation is conceptual rather than numerical. SelfJudge primarily studies how to build stable unlabeled training signals, not how to overcome the Judge’s own capability ceiling. The Judge is frozen, so sustained self-evolution is bottlenecked by judge quality; if both self-consistency and the Judge reinforce the same wrong answer, the framework can still amplify an incorrect consensus [2603.21289]. A related warning comes from JudgeBench: judging can be as hard as solving, and self-family evaluation can be especially difficult. In the generator-swap ablation, **Claude-3.5-Sonnet** scores **64.3%** on GPT-4o-generated pairs but only **44.8%** on Claude-generated pairs, indicating that a model can struggle most on the kinds of subtle errors its own family tends to produce [2410.12784].

Taken together, these results place SelfJudge in a precise position. It is a substantive method for fully unlabeled multimodal reasoning post-training, and a useful node in the larger theory of model-based evaluation. But it also belongs to a broader research program showing that self-judging is only reliable when bias control, prompt stability, metric choice, cross-mode consistency, and external validation are treated as first-class technical problems [2603.21289][2603.01865].

Source: https://www.emergentmind.com/topics/selfjudge