Virtual Judge Labeling Methods

• Virtual Judge labeling is a family of methods that generate labels via programmable judge mechanisms instead of direct human adjudication.
• It leverages techniques like annotator emulation, synthetic flaw injection, rubric-based scoring, and calibration to ensure label reliability.
• Applications span multi-turn dialogue evaluation, image safety, fairness auditing, and 3D preference tasks, though challenges include high computational costs.

Virtual Judge (VJ) labeling denotes a family of labeling procedures in which labels are produced by a configurable judge mechanism rather than by direct per-example human adjudication. In "Jury Learning," VJ labeling is defined explicitly: every human annotator in the training set is modeled by its own “juror” network, a user specifies a “judge panel” by group composition, the system samples neural models corresponding to real annotators who meet those criteria, and the panel’s aggregation is the label (Gordon et al., 2022). In adjacent LLM-as-a-judge and MLLM-as-a-judge work, the same paradigm expands to synthetic benchmark generation, constitution-based safety inspection, multimodal rubric scoring, multilingual degradation audits, and de-biased preference labeling for 3D generation, all of which use model-based judging pipelines to construct labels, rankings, or preference pairs at scale (Guerdan et al., 7 Mar 2025, Tang et al., 20 May 2026, Wang et al., 2024, Sahili et al., 26 Oct 2025, Asaria et al., 18 Jun 2026, KC, 21 Jun 2026).

1. Conceptual scope and major formulations

Within the literature, VJ labeling has two closely related technical meanings. The first is annotator emulation: a learned pool of per-annotator models is sampled to form a virtual panel whose majority, average, or weighted vote defines the label. The second is judge-system labeling: an LLM or MLLM is prompted, calibrated, or structurally constrained so that its outputs become the labels for downstream evaluation, safety inspection, or preference learning. The common thread is that the labeling operator is not a single fixed gold labeler, but a programmable judge mechanism with explicit assumptions about composition, aggregation, evidence, or calibration.

The supplied papers instantiate this design space in several distinct ways.

Formulation	Labeling mechanism	Representative source
Annotator-panel VJ labeling	Sample per-annotator neural models into a specified panel and aggregate their votes	(Gordon et al., 2022)
Synthetic multi-turn judge benchmarking	Generate paired conversations with one injected flaw and known better/worse side	(Tang et al., 20 May 2026)
Constitution-based zero-shot safety labeling	Objectified rules, precondition chains, relevance scan, debiased judgment, cascaded CoT	(Wang et al., 2024)
Multimodal rubric-based judging	Closed label sets, evidence grounding, abstention, rubric-mapped scores	(Sahili et al., 26 Oct 2025)
Reasoning-guided multimodal judge	Shared MLLM judge trained on a small text-reasoning corpus	(Ko et al., 24 May 2025)
De-biased 3D preference labeling	Swap-consistent VLM judging over quality-contrastive render pairs	(Asaria et al., 18 Jun 2026)
Multilingual reliability auditing	Gold-labelling by degradation with dual-order presentation	(KC, 21 Jun 2026)
No-gold-label validation	Response-set and distributional validation of judge systems	(Guerdan et al., 7 Mar 2025)

This suggests that “Virtual Judge” is less a single architecture than a methodological family organized around programmable adjudication, explicit aggregation, and scalable replacement or reduction of human labeling.

2. Annotator-model Virtual Judge labeling

In the Jury Learning formulation, the dataset consists of examples $X$ and $M$ annotators, each of whom has labeled some subset of $X$. Annotator $i$ has a labeled set $D_i \subset X \times Y$. The system learns one neural network $f_i(x;\theta,w_i)$ per annotator, with a shared feature extractor $\theta$ and annotator-specific head weights $w_i$. The total training loss is

$$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$

where $\mathcal R$ is a regularizer and $M$0 is a scalar (Gordon et al., 2022).

The network architecture uses a pre-trained transformer or CNN to embed $M$1 into $M$2, for example BERT’s [CLS] vector. Each annotator then receives a small MLP or single linear layer,

$M$3

with $M$4. The same source also permits low-dimensional group embeddings $M$5 and per-annotator residual embeddings $M$6, which can be concatenated with $M$7 and fed into a small Deep & Cross network (Gordon et al., 2022).

At inference time, the user specifies a panel composition. If there are $M$8 groups and composition vector $M$9, then for jury size $X$0, $X$1 seats are assigned to group $X$2. Writing $X$3, the system samples exactly $X$4 annotators from each $X$5, or equivalently defines per-annotator sampling probability

$X$6

The panel then predicts by majority vote for classification, by average for regression, or by weighted vote $X$7 (Gordon et al., 2022).

This construction makes disagreement a first-class object rather than collapsing it by majority vote during training. It also enables “counterfactual juries”: after training, one may ask which minimal change in $X$8 would flip the verdict by solving the quadratic program

$X$9

Uncertainty can be estimated by repeating jury sampling $i$0 times, collecting jury votes or means, and reporting the median and percentile bands; calibration may be improved by temperature-scaling or Platt scaling per annotator before aggregation (Gordon et al., 2022).

3. Label construction protocols in LLM- and MLLM-based systems

A major branch of VJ labeling constructs labels by design rather than by fitting annotator-specific heads. RankJudge is exemplary: it takes a corpus of reference documents, samples a reference document $i$1, a user behavior type $i$2, an assistant failure type $i$3, a flawed turn $i$4, and the worse side $i$5 or $i$6, then generates two plans, synthesizes two $i$7-turn dialogues, and applies a three-layer automated verification cascade consisting of coherence, adherence, and grounding checks (Tang et al., 20 May 2026). The retained item is a tuple $i$8, where $i$9 is the known better side. Judges receive only $D_i \subset X \times Y$0 and must output $D_i \subset X \times Y$1. Credit is given only under the strict joint correctness indicator

$D_i \subset X \times Y$2

Because the paired conversations differ by a single injected flaw, labels are known by construction and localized to one turn (Tang et al., 20 May 2026).

FairJudge defines a rubric-driven VJ protocol for prompt-image alignment and social attributes. For alignment, the judge outputs a discrete rating $D_i \subset X \times Y$3 with anchors from “Not match at all” to “Matches exactly,” then maps it linearly to $D_i \subset X \times Y$4. For social attributes, the judge is constrained to closed label sets for Gender, Race, Age, Religion, Culture, Disability, and optionally Profession; it must provide evidence grounded in visible content and output “unspecified” iff no unambiguous, directly observable visual cue supports any label in that field’s closed set (Sahili et al., 26 Oct 2025). The label construction mechanism therefore couples discrete taxonomies, visible-cue rationales, and principled abstention.

CLUE, the constitution-based zero-shot image safety pipeline, constructs labels from a safety constitution $D_i \subset X \times Y$5 rather than from human-labeled training examples. Rules are first “objectified” through an LLM-as-an-Optimizer loop until each rule achieves at least $D_i \subset X \times Y$6 objectiveness score. Each objectified rule is then decomposed into a logically complete chain of Boolean preconditions $D_i \subset X \times Y$7, and only if all preconditions hold is the rule violated (Wang et al., 2024). To reduce computation, CLIP-ViT cosine similarity filters out rules whose relevance is below threshold $D_i \subset X \times Y$8. Remaining preconditions are judged via debiased token probabilities, region-removal tests using OWLv2, and, for borderline cases, cascaded chain-of-thought reasoning that ends in a concise JSON summary. The final output is a safe/unsafe label together with violated rules (Wang et al., 2024).

BabelJudge adopts a different label-construction principle: gold-labelling by degradation. Starting from a high-quality reference response $D_i \subset X \times Y$9, it applies atomic perturbation functions such as truncate, shuffle, number_corrupt, drop_entities, and repeat_pad at severities $f_i(x;\theta,w_i)$0 to obtain a perturbed response $f_i(x;\theta,w_i)$1 that is guaranteed to be strictly worse than $f_i(x;\theta,w_i)$2 (KC, 21 Jun 2026). The item is then $f_i(x;\theta,w_i)$3, and each item is presented twice to the judge, once with the reference first and once with the perturbed response first. This makes the gold label known by construction while also exposing slot-order effects.

In the de-biased VLM-as-3D-judge protocol, preference pairs are likewise engineered rather than manually labeled. A single furniture image $f_i(x;\theta,w_i)$4 is encoded with DINOv2 features, TRELLIS is run twice to produce a high-quality pass $f_i(x;\theta,w_i)$5 and a low-quality pass $f_i(x;\theta,w_i)$6, both are rendered as $f_i(x;\theta,w_i)$7 normal-map montages with the reference image, and a training judge labels the winner and loser under a swap-consistency protocol (Asaria et al., 18 Jun 2026). The resulting preference pairs are then used for preference-based fine-tuning or conditioner repair.

4. Scoring, ranking, and difficulty estimation

Once labels are generated, VJ systems require a second layer of formalism to score judges, items, or panels. RankJudge compares $f_i(x;\theta,w_i)$8 candidate judges on $f_i(x;\theta,w_i)$9 retained items with a Bradley–Terry pairwise-ranking model. Each judge $\theta$0 has latent strength $\theta$1 and each item $\theta$2 has difficulty $\theta$3, with

$\theta$4

The parameters are fit by maximizing the log-likelihood over observed outcomes $\theta$5, then mean-normalized and transformed to Elo via $\theta$6; cluster-robust sandwich standard errors at the item level yield $\theta$7 confidence intervals (Tang et al., 20 May 2026). As a by-product, items receive difficulty scores, and pruning the top $\theta$8 most difficult items defines a cleaner “evaluation slice.”

RankJudge also reports an alternative PageRank-style random walk, Empirical Interaction Propagation (EIP), on the bipartite judge-item graph with damping $\theta$9. The paper prefers Bradley–Terry because it enjoys closed-form error bars and stability under partial observability (Tang et al., 20 May 2026). In the reported experiments, three domains produced $w_i$0 candidate pairs per domain, about $w_i$1–$w_i$2 were retained after verification, and $w_i$3 frontier judges were evaluated on a curated slice of about $w_i$4 items. Elo scores spanned nearly $w_i$5 points. Judge rankings remained stable when the failure-type component was dropped or pointwise Likert evaluation was used, with Spearman $w_i$6; re-fitting Bradley–Terry on only $w_i$7–$w_i$8 of matches retained $w_i$9; swapping to EIP yielded $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$0 (Tang et al., 20 May 2026).

FairJudge defines a different score family. Alignment receives $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$1. For each social attribute $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$2 with true label $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$3 and predicted label $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$4, the correctness score is $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$5 if $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$6, $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$7 if $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$8 and $$L(\theta,\{w_i\}) \;=\; \sum_{i=1}^{M}\;\sum_{(x,y)\in D_i} \ell\bigl(f_i(x;\theta,w_i),\,y\bigr)\;+\;\lambda\,\mathcal R(\theta,w_1,\dots,w_M),$$9, and $\mathcal R$0 if $\mathcal R$1. Averaging over the evaluated attribute set $\mathcal R$2 gives $\mathcal R$3, and an optional combined score is $\mathcal R$4 with $\mathcal R$5 (Sahili et al., 26 Oct 2025).

BabelJudge turns reliability into an explicit bias-penalized composite. It defines raw accuracy, position bias $\mathcal R$6, verbosity bias $\mathcal R$7, and order consistency $\mathcal R$8, then sets a competence base $\mathcal R$9 and a final reliability score

$M$00

For Qwen2.5-7B-Instruct-4bit, the reported text-benchmark reliability scores were $M$01 for English, $M$02 for Hindi, $M$03 for Arabic, and $M$04 for Swahili; raw accuracy alone understated this gap, with macro-accuracy $M$05 versus macro-$M$06 $M$07 (KC, 21 Jun 2026).

The 3D judging protocol uses win-rate and order-flip rate rather than Elo or rubric scores. Win-rate is the proportion of swap-consistent pairs on which the judge picks the “better” mesh, while flip-rate is the fraction of pairs where verdicts disagree between $M$08 and $M$09. Calibration tests show clear-gap win-rate approximately $M$10–$M$11, base-vs-base win-rate approximately $M$12, and base-vs-base flip-rate approximately $M$13 (Asaria et al., 18 Jun 2026).

5. Validation without gold labels

Theoretical analysis becomes necessary when labels cannot be assumed to be uniquely correct. Guerdan et al. formulate the VJ problem as judge validation in the absence of gold labels. Each item $M$14 has response options $M$15, and both humans and judges may be elicited either by forced choice (pick exactly one option) or by response set (endorse all reasonable options). A task is “underspecified” when $M$16, so multiple endorsements may be legitimate even though forced choice suppresses them (Guerdan et al., 7 Mar 2025).

The paper decomposes human ratings into a stable response-set distribution $M$17, an error operator $M$18, and a forced-choice selection operator $M$19, yielding

$M$20

and shows that $M$21 is recoverable from observed forced-choice distributions only if the rating task is fully specified (Guerdan et al., 7 Mar 2025). Judge performance is then defined by comparing aggregated judge ratings $M$22 and aggregated human ratings $M$23 under a scalar metric $M$24, with population-level performance $M$25. The paper further proves that if two definitions of performance are not monotone transforms of one another, there exists some distribution under which they rank judges in opposite order (Guerdan et al., 7 Mar 2025).

The empirical consequences are substantial. In synthetic experiments, one forced-choice rating per item on a fully specified task matches three ratings per item on an underspecified task, corresponding to a $M$26 budget saving. Categorical metrics such as Hit-Rate and $M$27 fail badly under asymmetric forced-choice selection bias. Even without a judge, one forced-choice rating per item misclassifies $M$28–$M$29 of items versus population consensus and underestimates prevalence (Guerdan et al., 7 Mar 2025). In the Civil Comments case study, the selected judge depended on the metric even at $M$30, and at $M$31, metrics that always selected Sonnet 3.5 yielded up to $M$32 lower consistency than the true winner Mistral Small (Guerdan et al., 7 Mar 2025).

Synthetic VJ pipelines can be read against this backdrop. RankJudge’s flaw injection and BabelJudge’s degradation design create labels whose correctness is known by construction rather than inferred from ambiguous human consensus (Tang et al., 20 May 2026, KC, 21 Jun 2026). This suggests that one function of synthetic VJ labeling is to move judge validation from indeterminate human aggregation toward mechanically specified label semantics, while still requiring separate calibration against label noise, ambiguity, or bias. RankJudge’s random $M$33-pair human audit found that label noise and ambiguity concentrated in the top difficulty tail, which motivated pruning the hardest $M$34 of items (Tang et al., 20 May 2026).

6. Biases, applications, and limitations

Bias auditing is central to contemporary VJ labeling. BabelJudge identifies four failure modes that raw accuracy can hide: position bias, verbosity bias, order inconsistency, and cross-lingual degradation (KC, 21 Jun 2026). In RankJudge, weaker judges overpredicted evasion by $M$35 percentage points and underpredicted no_clarification by $M$36 percentage points, indicating that taxonomy resolution itself is a capability gap (Tang et al., 20 May 2026). In the 3D setting, three render-side failure modes had to be corrected: image overload collapsed Qwen2.5-VL to “always pick left,” Gaussian-splat renders hid geometry defects, and reference-free judging rewarded clean-but-wrong geometry (Asaria et al., 18 Jun 2026). FairJudge addresses a related failure mode—taxonomy drift—by forcing a closed label set plus “unspecified” fallback and requiring rationales grounded in visible content (Sahili et al., 26 Oct 2025).

The application range is correspondingly broad. VJ labeling has been used for multi-turn, document-grounded conversational evaluation in machine learning, biomedicine, and finance (Tang et al., 20 May 2026); for online toxicity and other annotator-disagreement tasks through virtual juries (Gordon et al., 2022); for text-to-image alignment and fairness audits over gender, race, age, religion, culture, disability, and profession (Sahili et al., 26 Oct 2025); for zero-shot image safety inspection under an objectified safety constitution (Wang et al., 2024); for text, image, audio, video, and molecule judging via reasoning-based multimodal transfer (Ko et al., 24 May 2025); for single-image 3D mesh preference labeling and optimization (Asaria et al., 18 Jun 2026); and for multilingual and agentic evaluation through perturbation-based reliability auditing over English, Hindi, Arabic, and Swahili, as well as nine trajectory-level perturbations (KC, 21 Jun 2026).

The limitations are equally explicit. In Jury Learning, large $M$37 makes one head per annotator expensive, motivating clustering, low-dimensional identifiers, or hypernetworks; reliable group metadata are required; fairness constraints on $M$38 may be necessary; cold-start annotators require initialization; and inference cost scales with $M$39 forward passes (Gordon et al., 2022). Flex-Judge reports that more than $M$40K text examples in the seed-size sweep led to catastrophic forgetting on image tasks, even though the model generalized broadly across modalities when trained on minimal textual reasoning data (Ko et al., 24 May 2025). The de-biased 3D study reports only directional results at $M$41 objects, evaluates only parameter-efficient adaptation, focuses on TRELLIS and furniture, uses synthetic degradations, and includes no large-scale human judgments (Asaria et al., 18 Jun 2026). The no-gold-label validation framework assumes rater error independent of rater identity and does not model rater correlation or item-specific error (Guerdan et al., 7 Mar 2025).

Across these formulations, the defining property of VJ labeling is not merely automation, but explicit control over how labels are produced: by sampled annotator panels, by synthetic flaw injection, by constitutions and preconditions, by evidence-grounded rubrics with abstention, by degradation with dual-order probing, or by de-biased cross-model preference protocols. The literature therefore treats VJ labeling as a general mechanism for scalable adjudication under disagreement, multimodality, and limited human annotation budgets, with reliability determined as much by task specification, aggregation, and calibration as by judge model capacity.