Dual-Judge Evaluation Approach

• Dual-judge evaluation approach is a framework where two LLM-based judge agents use distinct personas to assess NLG outputs with multidimensional criteria.
• It employs a structured process of independent scoring, rebuttal rounds, and score aggregation to simulate expert consensus and capture diverse perspectives.
• Empirical results demonstrate enhanced reliability and correlation with human ratings, achieving improvements in domains such as education and medicine.

The dual-judge evaluation approach refers to an automated framework in which two independent judge agents—typically instantiated as LLMs adopting distinct, well-defined evaluation personas—are employed to assess, discuss, and collectively score the quality of a target natural language generation (NLG) output, such as a student answer or dialog response. This methodology enables more robust, multidimensional, and human-aligned assessment than single-judge protocols, while remaining computationally tractable, and serves as the foundational case of broader multi-judge distillation and debate-based judging systems (Chen et al., 28 Jul 2025, Tang et al., 1 Aug 2025).

1. Rationale and Theoretical Foundations

The need for dual-judge (or multi-judge) systems arises from limitations of the LLM-as-a-judge paradigm, where a single LLM, typically prompted generically or with a simple persona, provides evaluation signals for NLG outputs. Single-judge approaches are prone to arbitrary or inconsistent persona design and often lack generalizability across domains. Most importantly, they fail to capture the multi-dimensionality and perspective diversity inherent in human expert evaluation, especially within high-stakes settings (e.g., education, medicine), where real experts often disagree or prioritize different criteria (Chen et al., 28 Jul 2025).

Dual-judge frameworks leverage the idea that collaborative “many-minds” evaluation more faithfully simulates human review panels, improves coverage over evaluation criteria, exposes consensus versus dissent, and—when structured carefully—improves correlation with expert ground truth. Moreover, by selecting two differentiated, core perspectives, one can achieve much of the robustness and reliability boost associated with full-scale multi-agent systems while incurring only modest computational cost (Tang et al., 1 Aug 2025).

2. Persona and Evaluation Dimension Construction

The dual-judge pipeline begins by grounding judge personas in domain-relevant documentation—guidelines, qualitative research, or best-practices papers representative of the application domain. The steps are as follows (Chen et al., 28 Jul 2025):

1. Domain Document Selection: Choose 2–5 documents (e.g., clinical-practice guidelines; teacher interviews).
2. Dimension Extraction via LLM: Employ an LLM-based stakeholder-identification prompt at the paragraph level to extract tuples—stakeholder name, evaluation dimensions (e.g., “pedagogical clarity,” “clinical consistency”), and supporting evidence quotes.
3. Clustering and Filtering: Cluster similar stakeholders (e.g., “primary school teacher” and “tutor” into “Educator”) using embeddings or LLMs, and select exactly two core, contrasting perspectives. Each receives 2–4 associated dimensions (e.g., Accuracy, Fluency, Relevance for education; Factuality, Clarity, Evidence Alignment for medicine).
4. Persona Instantiation: For each perspective, construct a JSON-style persona prompt specifying realistic biographical information, evaluation mission (“I focus on whether questions promote creative thinking beyond factual recall”), domain specialty, selected personality traits, and a social relationship with other stakeholders (see Table 1).

Persona Name	Perspective Domain	Key Dimensions
Emma Porter	Early childhood teacher	Accuracy, Fluency, Relevance
Dr. Wang	Clinician	Factuality, Clarity, EvidenceAlignment

Table 1: Example judge personas and dimensions for dual-judge evaluation (Chen et al., 28 Jul 2025).

3. Structured Dual-Judge Scoring and Debate Protocol

The evaluation protocol is realized in three structured phases (Chen et al., 28 Jul 2025):

• Phase 1: Independent Scoring Each judge, prompted with their persona, independently rates the NLG output, returning a JSON structure with per-dimension scores (1–5 scale) and concise, dimension-specific commentaries.
• Phase 2: Structured Rebuttal Rounds Judges engage in turn-based rebuttal, with a typical order of A → B → A → B, up to a maximum R rounds (R=3 typical). In each turn, the active judge references the other’s critique or score, either agreeing/amplifying or disagreeing/rebutting (<100 words), and may revise scores. The process terminates when both judges output “NO_MORE_COMMENTS” or R is reached.
• Phase 3: Final Judgment Announcement Both judges output final revised scoring JSONs and a brief summary sentence, enabling precise traceability of evaluation and changes.

This protocol exposes cross-perspective conflict and agreement, provides an explicit path for refining judgments, and creates a record that supports both quantitative and qualitative synthesis.

4. Aggregation, Agreement Measurement, and Output Synthesis

Post-debate, the outputs are aggregated along several axes (Chen et al., 28 Jul 2025):

• Numeric Aggregation: Per-dimension scores are normalized, and a weighted average $\big(w_A \cdot score_A' + w_B \cdot score_B'\big)$, with typically $w_A = w_B = 0.5$, yields the final metric.
• Inter-Judge Agreement: Reliability is assessed via Cohen's κ, calculated on the binned judgment ratings: $\kappa = \frac{p_o - p_e}{1 - p_e}$, where $p_o$ is observed agreement, and $p_e$ the chance agreement baseline determined by the marginal rating distributions.
• Qualitative Synthesis: Both commentaries are concatenated and, optionally, a meta-judge prompt is run to summarize points of consensus and dissent.

5. Distillation and Efficient Dual-Judge Models

Emerging work demonstrates that the outputs of dual-judge protocols can be distilled into compact automatic evaluators, reducing evaluation time without sacrificing fidelity. One approach employs a fixed LLM-based encoder $g$ to embed input dialogues, with a small MLP head $f_\omega$ producing a scalar quality score $s_\theta(x)$. Training leverages dual-judge-produced pairwise comparison datasets, employing a Thurstone Case V model and maximum likelihood objective that explicitly models judge noise parameters ($\alpha^j, \beta^j$) (Tang et al., 1 Aug 2025). This protocol enables aggregation of two judges’ subjective preferences into an implicit ground-truth distribution without naive majority voting.

At inference, this distilled model achieves a 15x speedup over two full LLM calls with negligible performance loss and improves Pearson correlation with human preference (e.g., increasing from ≈0.587 for single-judge, to ≈0.602 for dual, on xDial-IEval) (Tang et al., 1 Aug 2025).

6. Empirical Results and Domain Considerations

Experiments demonstrate that dual-judge setups consistently increase alignment with human expert ratings by +0.08–0.15 Spearman ρ over single-judge baselines and improve Cohen’s κ from ~0.25 to ~0.55 after a single rebuttal round in domains such as education and medical summarization (Chen et al., 28 Jul 2025). Domain prioritization—such as weighting engagement and developmental appropriateness in education judges, or factual consistency and evidence strength in clinical judges—can be flexibly realized by tuning dimension selection, persona prompts, and aggregation weights.

7. Significance and Implications

Dual-judge evaluation frameworks synthesize much of the reliability, multidimensionality, and self-correcting debate of more expansive multi-agent systems, yet remain practical for contemporary evaluation pipelines. Modeling judge-specific reliabilities and distilling “collective judgment” into efficient deployable models preserves most of the gains in correlation and robustness, providing a scalable alternative to traditional human and single-agent evaluations.

This suggests that even limited-agent collaborative evaluation, grounded in principled persona construction and structured debate, yields substantial empirically validated improvements in reliability, interpretability, and alignment with expert ground truth over both naive LLM-as-a-judge and single-metric automated approaches (Chen et al., 28 Jul 2025, Tang et al., 1 Aug 2025).