Judgement-Guided Label Verification (JGLV)

• JGLV is a framework that uses iterative, model-guided judgments to verify and purify noisy labels in large, synthetic, or biased datasets.
• It employs closed-loop methodologies and dependence-aware aggregation to prioritize error correction and improve downstream task performance.
• Empirical studies demonstrate that JGLV can yield significant accuracy gains and speed improvements in LLM tool-use, visual label cleansing, and robustness verification.

Judgement-Guided Label Verification (JGLV) refers to a suite of algorithmic frameworks and workflows in which model-generated judgments, often in conjunction with pretrained classifiers, dedicated LLM-based judges, or interactive human feedback, are used to assess, aggregate, and refine the correctness of labels within large, noisy, or synthetically generated datasets. JGLV has become a central methodology in applications ranging from LLM tool-use self-evolution, label purification in vision and text tasks, dependence-aware judgement aggregation, to formal verification of deep learning models. Core to JGLV is a closed-loop or iterative pipeline where the verification of annotations is not static, but adaptively guided by model or collective judgment, producing increasingly purified, high-value datasets and robust evaluation criteria.

1. Core Principles and Motivations

JGLV addresses the limitations of static or naively aggregated annotations, especially in settings where:

• Synthetic or automatically generated labels are noisy or may encode systematic biases (Zhang et al., 12 Nov 2025).
• Multiple annotators or models (e.g., LLMs-as-judges) provide correlated, non-independent signals (Balasubramanian et al., 29 Jan 2026, Guerdan et al., 7 Mar 2025).
• Verification or robustness analysis is computationally intensive and may benefit from decomposed, judgment-prioritized evaluation (Wan et al., 2020).
• Human-in-the-loop error correction is feasible and beneficial, but must be prioritized due to annotation costs (Bäuerle et al., 2018).

The central purpose of JGLV is to align the labeling process with the actual capability frontiers of current models or validation collectives, to dynamically correct errors—both in supervision and evaluation—without introducing excessive computation or annotation costs, and to achieve superior downstream model performance and evaluation reliability.

2. Representative Algorithmic Methodologies

JGLV takes a variety of algorithmic forms depending on the domain and application scenario:

A. Closed-Loop Dataset Purification (LLM Tool-Use)

Within the LoopTool framework, JGLV operates as a fully automated, model-aware label verification process. After an initial round where model predictions disagree with reference calls, each discordant example is passed to a powerful open-source LLM "judge" (Qwen3-32B) for adjudication. The judge returns one of four outcomes: PRED_WRONG, LABEL_WRONG, BOTH_CORRECT, BOTH_WRONG. Depending on the outcome:

• Retain example for retraining if the model is deemed wrong.
• Replace the noisy reference with the model's superior prediction if the label is wrong.
• Discard pair if both are incorrect; keep only hard boundary cases for further training if both are correct.

Mathematically, the key error seed subsets at iteration $j$ are: $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$ The merged set $\mathcal{D}_j^{ES}$ seeds downstream expansion and retraining, incrementally purifying the corpus (Zhang et al., 12 Nov 2025).

B. Dependence-Aware Judgment Aggregation

Classic aggregation methods (majority vote, Dawid-Skene) assume conditional independence, which fails for highly correlated LLM judges. JGLV can be instantiated as class-dependent Ising models: $$P(J|Y=y) \propto \exp\left[\sum_{j=1}^K h_j^{(y)}J_j + \frac{1}{2}\sum_{j\neq k} W_{jk}^{(y)}J_jJ_k\right]$$ where $J$ denotes binary judgments, $h_j^{(y)}$ are fields, $W_{jk}^{(y)}$ pairwise couplings. The Bayes decision is typically quadratic: $$g^*(J) = 1\left\{b_0 + \sum_j a_j J_j + \sum_{j<k} b_{jk}J_j J_k \geq 0 \right\}$$ Efficient expectation-maximization (EM) and pseudolikelihood algorithms allow for parameter fitting. Empirically, such dependence-aware JGLV yields up to 10–12% absolute accuracy gains over uniform voting (Balasubramanian et al., 29 Jan 2026).

C. Classifier-Guided Visual Correction

In vision tasks, JGLV instantiates as interactive correction loops guided by classifier predictions and quantitative error scores: CIES (class-level confusion), IIES (instance-level mislabel), SES (similarity/duplication). Samples are prioritized, visualized as confusion-matrix cells and embedded projections, and then presented to an annotator, who takes corrective actions (confirm, relabel, delete). This loop enables high recall of genuine noise with minimal inspection burden (Bäuerle et al., 2018).

D. Robustness Verification in Deep Learning

JGLV is used to decompose monolithic robustness verification into per-target-label subproblems. For a feed-forward ReLU network and input $x$ with true label $y$, checking robustness amounts to verifying for each $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$0,

$$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$1

with symbolic interval propagation and linear relaxation heuristics to prioritize $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$2. This decomposition yields $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$3 to $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$4 practical speedup over monolithic approaches (Wan et al., 2020).

3. Workflow Integration and Practical Design

A common pattern in JGLV frameworks is the integration with upstream diagnostics and downstream data synthesis or retraining:

• Initial capability probing (e.g., GCP in LoopTool) flags discordant instances.
• JGLV either acts via automated judgment, visualized human review, or collective aggregation to purify/correct/discard examples.
• The purified error seeds drive further data augmentation or error-focused retraining.
• The loop repeats, with the purified dataset improving both label quality and downstream model generalization or robustness (Zhang et al., 12 Nov 2025, Bäuerle et al., 2018).

Computationally, JGLV can be efficiently implemented with a single LLM judge (as in LoopTool), per-instance user interaction (as in visual correction), or scalable aggregation methods (Ising/pseudolikelihood). Avoiding overcorrection and feedback loops that reinforce model errors is managed by strictly gating label replacement and discarding ambiguous or clearly flawed samples.

4. Theoretical Underpinnings and Limitations

JGLV efficacy relies on both the power and calibration of the judge or classifier and the soundness of aggregation or subdivision strategies. In contexts with ambiguous or underspecified labels, as shown in the absence-of-gold-label analysis, naive forced-choice or single-metric judge selection can yield significant degradation—up to 34% underperformance—relative to properly modeled, multi-label, or distributional JGLV procedures (Guerdan et al., 7 Mar 2025).

The theoretical analysis of rating aggregation tasks within JGLV exposes:

• The necessity of rating task specification (full response set coverage $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$5).
• The instability of categorical metrics (hit rate, Cohen’s $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$6) under asymmetric selection effects.
• The importance of dependence modeling (Ising/latent factor) for LLM-judge aggregation (Balasubramanian et al., 29 Jan 2026).

Soundness of JGLV-based robustness verification is formally guaranteed provided the underlying verifier is correct; otherwise, any found adversarial example is genuine (Wan et al., 2020).

5. Empirical Performance and Impact

Empirical studies demonstrate that JGLV frameworks substantially improve both data and model performance:

• LoopTool's removal of JGLV drops BFCL-v3 accuracy by up to 1.73 points in ablation, and inclusion leads to new state-of-the-art tool-use performance, surpassing the data generator by over 8 points (Zhang et al., 12 Nov 2025).
• Classifier-guided interactive loops on vision data recover over 85% of synthetic mislabels in 15 minutes, with >4% accuracy gain after retraining (Bäuerle et al., 2018).
• Dependence-aware Ising aggregation for LLM judges delivers 6–8% absolute gains over conditional independence models and greater as judge count and training size increase (Balasubramanian et al., 29 Jan 2026).
• Per-label JGLV for DNN robustness speeds up verification by up to $$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$7, especially when counterexamples exist (Wan et al., 2020).

Aggregate metrics from recent work are summarized:

Scenario	Baseline Method	JGLV-Enabled Metric	Absolute Improvement	Reference
LLM tool-use learning	Static pipeline	Closed-loop (JGLV)	+8.59 points (BFCL-v3 acc.)	(Zhang et al., 12 Nov 2025)
LLM judge aggregation	Majority/CI vote	CD-Ising	+6–8% (test accuracy)	(Balasubramanian et al., 29 Jan 2026)
Image label cleansing	Manual/naive	Guided visual JGLV	+4.7% (MNIST accuracy)	(Bäuerle et al., 2018)
DNN robustness	Monolithic verify	Per-label JGLV	$$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$8–$$\mathcal{D}_j^{PW} = \{(\mathcal{T},c_t,a_t^*;a_t)| y_\text{judge} = \text{PRED\_WRONG}\}, \quad \mathcal{D}_j^{LW} = \{(\mathcal{T},c_t,a_t := a_t)| y_\text{judge} = \text{LABEL\_WRONG}\}.$$9 speedup	(Wan et al., 2020)

These results establish JGLV as a crucial component in modern data-centric and evaluation pipelines in deep learning and LLM development.

6. Extensions, Recommendations, and Generalization

JGLV frameworks are broadly applicable and model-agnostic, provided the existence of:

• A mechanism for generating or collecting alternative judgments (model, LLM, human, or classifier output).
• A diagnostic signal regarding disagreement or error likelihood (error types, confusion, uncertainty).
• A procedure for corrective action (relabeling, aggregation, discarding).

Recommended best practices from empirical and theoretical analyses include:

• Fully specify rating schemes to maximize interpretability and identifiability (Guerdan et al., 7 Mar 2025).
• In multi-judge settings, model dependency structures explicitly, avoiding conditional independence assumptions in aggregation (Balasubramanian et al., 29 Jan 2026).
• For highest throughput and correction accuracy, combine automated JGLV with human-in-the-loop review prioritized by error scores (Bäuerle et al., 2018).
• For computationally intensive verification, decompose problems under JGLV-guided prioritization (Wan et al., 2020).

JGLV continues to evolve as a unifying principle bridging model-led, human-in-the-loop, and statistical judgment aggregation for data and evaluation purification in machine learning.