SelfJudge: Unsupervised LLM Self-Evolution

• SelfJudge is a framework for unsupervised self-evolution in LLMs, combining self-consistency signals with integrated judge modulation to refine outputs.
• It features a dual-role design where an actor samples multiple reasoning trajectories while a frozen judge evaluates answer correctness, reasoning quality, and visual grounding.
• The architecture employs bounded judge modulation and group-relative policy optimization, enhancing performance on multimodal tasks without relying on human annotations.

“SelfJudge” denotes two related but non-equivalent ideas in contemporary LLM evaluation. In the narrow sense, it refers to the unsupervised multimodal self-evolution framework in which one model instantiates an Actor and a frozen Judge, samples multiple reasoning trajectories on unlabeled image–question inputs, and improves itself using self-consistency, bounded judge modulation, and group-relative policy optimization (Wu et al., 22 Mar 2026). In a broader sense, the term overlaps with the wider design space of LLM-as-a-judge systems, including same-family evaluators, self-improving judges, and externalized judge models used to score or compare model outputs (Li et al., 2024, Liu et al., 26 Feb 2025). The distinction is consequential: some systems study a model’s own evaluation of its generations, whereas others study standalone evaluator models whose relevance to “self-judging” is indirect but substantial.

1. Conceptual scope

In strict usage, SelfJudge is not merely “an LLM used as a judge,” but a training architecture in which the model improves on unlabeled data through internally generated supervisory signals (Wu et al., 22 Mar 2026). The broader literature, however, often uses closely related language for any judge model that evaluates model outputs, including rubric-conditioned open judges, reward-model-like evaluators, or role-split judge agents. This broader usage is technically useful but conceptually looser.

That distinction is explicit in the surrounding literature. EasyJudge, for example, is best understood as an open-source LLM-as-a-judge system for evaluating LLM outputs, not as a strict self-judging method; it evaluates responses from other models or arbitrary candidate responses rather than a model explicitly critiquing its own output (Li et al., 2024). By contrast, JAF uses the same underlying LLM family for both the primary agent and the judge agent, with different prompts implementing the distinct roles, which places it closer to role-prompted self-judging and iterative self-refinement (Garg et al., 29 Jan 2026).

The broader SelfJudge landscape is therefore heterogeneous. It includes: a narrow self-evolution framework over unlabeled multimodal data; cohort-aware self-refinement systems; cross-prompt self-improving judges; post-hoc calibrated judges; and externalized evaluator platforms. A technically precise account must keep these senses separate.

2. SelfJudge as unsupervised self-evolution

In the narrow formulation, SelfJudge addresses multimodal reasoning post-training without human-annotated answers, external reward models, verifier labels, or teacher-model distillation (Wu et al., 22 Mar 2026). For each unlabeled input $x=(I,q)$, the Actor policy $\pi_\theta$ samples a group of reasoning trajectories,

$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$

and each trajectory $\tau_i$ ends in a final answer $a_i$.

The first training signal is the Actor’s own self-consistency distribution. If $\hat p(a)$ is the empirical frequency of answer $a$ in the sampled group, then each trajectory receives the soft prior

$r_i^{\mathrm{SC}} = \hat p(a_i).$

This departs from majority-vote pseudo-labeling, which would collapse the group to a single answer. The paper’s claim is that soft self-consistency preserves more within-group structure than hard majority reward and avoids amplifying an early dominant mode as aggressively (Wu et al., 22 Mar 2026).

A second role is played by the Judge. The Judge is not an external model; it is initialized as a structurally identical copy of the current Actor at the start of training and then frozen. Given input $x$ and trajectory $\tau_k$, it outputs a score

$\pi_\theta$0

According to the appendix prompt, this score aggregates answer correctness, reasoning quality, and visual grounding, with weights $\pi_\theta$1, $\pi_\theta$2, and $\pi_\theta$3, respectively; a mandatory validity rule assigns all-zero scores if the response lacks a clear single final answer in the required format (Wu et al., 22 Mar 2026).

This architecture makes SelfJudge neither pure self-consistency training nor conventional reward-model RL. It is a closed-loop system in which the Actor supplies a low-variance prior from answer agreement, while the frozen Judge supplies a quality-sensitive correction signal.

3. Reward shaping, distributional modeling, and GRPO

SelfJudge does not use the raw Judge score directly as reward. Instead, the Judge modulates the self-consistency prior through a bounded function

$\pi_\theta$4

chosen to be continuous, bounded, and centered around $\pi_\theta$5 so that high-quality trajectories are amplified and low-quality trajectories are suppressed without letting the Judge completely override the Actor prior (Wu et al., 22 Mar 2026). The final reward is

$\pi_\theta$6

where $\pi_\theta$7 marks output-format violations and $\pi_\theta$8.

The method then converts absolute rewards into a within-group distribution. With $\pi_\theta$9,

$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$0

This group-relative construction is central. The paper argues that absolute Judge scores are not comparable across inputs, whereas relative advantages inside the sampled trajectory set are more stable and reduce collapse (Wu et al., 22 Mar 2026).

Optimization uses Group Relative Policy Optimization (GRPO) with clipping and KL regularization. In the main experiments, both Actor and Judge start from Qwen2.5-VL-7B-Instruct; training uses 8 trajectories per question, 20 epochs, AdamW, one node with $$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$1 NVIDIA A800 80GB GPUs, learning rate $$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$2, KL coefficient $$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$3, $$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$4, and reward-modulation hyperparameters

$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$5

(Wu et al., 22 Mar 2026). The Actor prompt enforces reasoning inside > ... and a single final answer in \boxed{ANSWER}.

The unlabeled training data come from Geometry3K (Geo3K), GeoQA, and MMR1, with labels discarded. Evaluation is reported on MathVision, MathVerse, WeMath, LogicVista, and DynaMath, with additional checks on ChartQA and MMVP (Wu et al., 22 Mar 2026).

4. Related self-judging and judge-improvement paradigms

The narrow SelfJudge framework sits inside a broader family of judge architectures. JAF turns self-judging into a cohort-level process: a judge evaluates each query–response pair together with a small neighborhood of peer examples, inducing a graph over the cohort and enabling critique propagation across related instances (Garg et al., 29 Jan 2026). In the reported experiments, both the primary agent and the judge use Llama3.3-Nemotron-Super-49B-v1.5 with different prompts, and repeated randomized judging yields an empirical acceptance probability

$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$6

which functions as a stability signal for self-evaluation (Garg et al., 29 Jan 2026).

ConsJudge addresses a different failure mode: prompt-sensitive judge instability in RAG evaluation. It generates judgments under $$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$7 hybrid aspect prompts built from hallucination, completeness, coherence, and semantic consistency, embeds those judgments, and chooses the most consensus-like and least consensus-like outputs as preference pairs for DPO training (Liu et al., 26 Feb 2025). This is not strict self-judging either, but it is a direct method for improving a judge using its own cross-prompt agreement structure.

JudgeLRM pushes the same direction from the optimization side. It treats judging as a reasoning-intensive task and trains judge models with GRPO and outcome-driven rewards over structured <think> and <answer> outputs (Chen et al., 31 Mar 2025). On PandaLM, JudgeLRM-7B, JudgeLRM-8B, and JudgeLRM-14B surpass DeepSeek-R1 by more than 2% in F1, while JudgeLRM-3B and JudgeLRM-4B exceed GPT-4 on the same benchmark (Chen et al., 31 Mar 2025). The implication for SelfJudge is that evaluator quality depends strongly on reasoning-oriented optimization, not only on evaluator prompting.

Flex-Judge generalizes the judge-building problem across modalities. It shows that 1K text-only reasoning examples can train a multimodal judge that transfers to image, video, audio, and molecule evaluation, and that the resulting judge can support best-of-$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$8 selection and DPO preference generation in the molecular setting (Ko et al., 24 May 2025). This is closely aligned with SelfJudge as a practical recipe for self-reward or same-family judge construction.

By contrast, EasyJudge represents the externalized end of the design space: an open-source, pairwise/pointwise, rubric-conditioned evaluator with a Streamlit-based web GUI, visualization, and auxiliary reference-based metrics, but not a method in which a model evaluates itself as part of an intrinsic training loop (Li et al., 2024).

5. Reliability, bias, and judge evaluation methodology

A major result of the wider literature is that judge reliability cannot be assumed, even when the judge is strong. JudgeSense formalizes prompt sensitivity through the Judge Sensitivity Score (JSS),

$$\mathcal{T}(x) = \{\tau_i\}_{i=1}^n, \quad \tau_i \sim \pi_\theta(\cdot \mid x),$$9

the fraction of paraphrase pairs on which a judge returns the same decision (Bellibatlu, 26 Apr 2026). On coherence, JSS ranges from 0.389 to 0.992 across models. On factuality, judges cluster near 0.63 in raw form because of a polarity-inverted prompt artifact; after correction, factuality JSS rises to about 0.9 or higher. In pairwise preference and relevance, 8 of 9 judges collapse into always-A behavior, making naive perfect consistency largely meaningless (Bellibatlu, 26 Apr 2026).

CyclicJudge studies a different pathology: persistent judge-specific bias. Its variance decomposition,

$\tau_i$0

shows that judge bias does not vanish by increasing the number of scenarios or sampled generations; it vanishes only by increasing the number of distinct judges or balancing them by design (Zhu et al., 2 Mar 2026). On MT-Bench, the paper reports concrete self-preference effects: under Qwen-as-judge, Qwen ranks itself first even though the paper says it is actually the weakest of the three evaluated models by the consensus of the other judges (Zhu et al., 2 Mar 2026). For SelfJudge, this is direct evidence that self-evaluation can materially reverse rankings.

Metric choice is also part of judge methodology. For prevalence-estimation tasks such as pass rate, refusal rate, or policy-violation rate, the paper on Balanced Accuracy argues that the right criterion for choosing a judge is Youden’s $\tau_i$1, or equivalently Balanced Accuracy, because judged prevalence obeys

$\tau_i$2

so the preservation of true model differences depends on $\tau_i$3, not on raw Accuracy or F1 (Collot et al., 8 Dec 2025). In parallel, LLM Jury-on-Demand treats reliability as instance-dependent and learns per-judge reliability predictors; on RAG groundedness it reaches 0.68 ± 0.02 Kendall’s Tau versus 0.58 ± 0.02 for static Average-All, and on RAG completeness 0.50 ± 0.03 versus 0.38 ± 0.03 (Li et al., 1 Dec 2025).

FairJudge integrates many of these concerns into training itself. It models judging as a policy

$\tau_i$4

constructs FairJudge-16K plus FairJudge-Benchmark-1K, and uses a curriculum SFT-DPO-GRPO pipeline to target rubric adherence, debiasing, and cross-mode consistency (Yang et al., 6 Feb 2026). This line of work suggests that SelfJudge is increasingly less about a single judge prompt and more about explicit control over judging behavior.

6. Empirical standing, applications, and limits

On its own core task, SelfJudge reports consistent gains over majority-vote self-training. With Geo3K as unlabeled training data, average accuracy across the five mathematical reasoning benchmarks rises from 34.6 to 37.9, and MathVision rises from 25.0 to 30.9 (Wu et al., 22 Mar 2026). Compared with MM-UPT, SelfJudge is consistently stronger: with Geo3K training, MM-UPT averages 35.4 while SelfJudge reaches 37.9; on MathVision the corresponding numbers are 27.5 and 30.9, and on DynaMath 21.4 and 24.2 (Wu et al., 22 Mar 2026).

The ablations isolate the method’s ingredients. Starting from Qwen2.5-VL-7B, the paper reports on MathVision / DynaMath: - base 25.0 / 20.3; - + Major Vote: 27.5 / 21.4; - + Self-Consistency: 25.2 / 20.5; - + Judge Scoring: 27.3 / 21.1; - + MV + JS: 28.4 / 22.7; - + SC + JS: 30.1 / 23.7; - + SC + JS (Dist.): 30.9 / 24.2 (Wu et al., 22 Mar 2026). The interpretation is straightforward: self-consistency alone is weak, judge scoring alone is better but unstable, and the combination with distributional modeling yields the strongest results.

The framework also shows some broader generalization. Under MMR1 training, the base model scores 85.7 / 77.2 on ChartQA / MMVP, MM-UPT gives 85.4 / 74.6, and SelfJudge gives 87.2 / 79.8 (Wu et al., 22 Mar 2026). At the same time, the paper explicitly identifies a trade-off in sample diversity: on MathVision / DynaMath, pass@10 drops from 0.66 / 0.48 in the base model to 0.64 / 0.43 under SelfJudge, though this is still stronger than majority vote and slightly better than supervised GRPO on average (Wu et al., 22 Mar 2026).

The sharpest limitation is conceptual rather than numerical. SelfJudge primarily studies how to build stable unlabeled training signals, not how to overcome the Judge’s own capability ceiling. The Judge is frozen, so sustained self-evolution is bottlenecked by judge quality; if both self-consistency and the Judge reinforce the same wrong answer, the framework can still amplify an incorrect consensus (Wu et al., 22 Mar 2026). A related warning comes from JudgeBench: judging can be as hard as solving, and self-family evaluation can be especially difficult. In the generator-swap ablation, Claude-3.5-Sonnet scores 64.3% on GPT-4o-generated pairs but only 44.8% on Claude-generated pairs, indicating that a model can struggle most on the kinds of subtle errors its own family tends to produce (Tan et al., 2024).

Taken together, these results place SelfJudge in a precise position. It is a substantive method for fully unlabeled multimodal reasoning post-training, and a useful node in the larger theory of model-based evaluation. But it also belongs to a broader research program showing that self-judging is only reliable when bias control, prompt stability, metric choice, cross-mode consistency, and external validation are treated as first-class technical problems (Wu et al., 22 Mar 2026, Zhu et al., 2 Mar 2026).