LLM-Jury Evaluation Method

• The paper presents a rigorous evaluation framework using LLM panels alongside SME inputs to measure alignment and statistical reliability.
• The methodology employs mixed-method pairwise comparisons with metrics such as percentage agreement, Cohen’s κ, and Pearson correlation.
• Findings reveal domain-specific biases and limitations, emphasizing the need for SME-in-the-loop validation in high-stakes evaluations.

The LLM-Jury Evaluation Method is a comprehensive framework for using LLMs as panels or "juries" to evaluate outputs generated by LLMs themselves, with an explicit emphasis on alignment with human (especially subject-matter expert, SME) judgment. This approach, as articulated in recent expert knowledge task studies, formalizes both the experimental workflow and the suite of statistical metrics necessary to rigorously assess the validity and reliability of LLM-based evaluation in knowledge-intensive and high-stakes domains (Szymanski et al., 26 Oct 2024).

1. Mixed-Methods Pairwise Comparison: Experimental Design

The LLM-Jury Evaluation Method operationalizes model evaluation by replicating human comparative judgment structures—most commonly, pairwise comparison. For each real-world prompt representing a domain-relevant instruction (e.g., from clinical, dietetic, or psychological guidelines), two candidate model outputs are generated using strong LLMs (e.g., GPT-4o vs. GPT-3.5-turbo) at a fixed decoding temperature.

A panel of LLM judges is then tasked to review both outputs per prompt, using a prompt that solicits both an overall preference and detailed, domain-specific aspect evaluations (such as accuracy, clarity, adherence to professional standards, personalization), returning both a discrete selection (A/B) and brief supporting rationale. Conditions are varied to include a "generic" judge persona and an "expert" persona (explicitly prompting the LLM to simulate a domain professional).

In parallel, SME panels (e.g., registered dietitians, clinical psychologists) perform the same evaluations using the same prompts and output pairs, with the addition of free-text rationales. Lay participants may also be included as a baseline for non-expert alignment. Randomization of prompt order and candidate position eliminates positional bias (Szymanski et al., 26 Oct 2024).

2. Evaluation Metrics and Mathematical Formulation

Robust assessment of jury reliability requires multiple statistical measures:

• Percentage Agreement: $P = \frac{\text{\# agreements}}{N} \times 100\%$
• Cohen’s Kappa: $\kappa = \frac{P_o - P_e}{1 - P_e}$, where $P_o$ is observed fraction agreement and $P_e$ is expected chance agreement.
• Pearson Correlation Coefficient: Used if fine-grained or scalar scores are present:

$$r = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum_i (x_i - \bar x)^2} \sqrt{\sum_i (y_i - \bar y)^2}}$$

The default is percent agreement, but Cohen’s κ is required for adjusting for chance agreement, and Pearson’s r is critical when continuous or ordinal scores are elicited rather than binary preferences. All LLM-SME and SME-SME judgements should be compared on these axes (Szymanski et al., 26 Oct 2024).

3. Empirical Results: Human-LLM Alignment by Domain and Aspect

The method reveals that LLM–SME agreement on overall preference only approaches 68% in dietetics (generic persona: 64%, expert persona: 68%) and 64% in mental health (generic: 60%, expert: 64%). Human–human (inter-expert) agreement is higher: 75% in dietetics and 72% in mental health.

Aspect-specific agreement varies sharply; for example:

Aspect	Dietetics (Gen→Expert)	Mental Health (Gen→Expert)
Clarity	55% → 60%	70% → 40%
Accuracy	56% → 67%	80% → 80%
Professional Standards	80% → 80%	64% → 73%
Educational Context	55% → 45%	60% → 70%
Personalization	56% → 44%	67% → 67%
Overall Preference	64% → 68%	60% → 64%

In control experiments with lay users, LLM-lay agreement rose to 80%, exceeding SME–LLM alignment, and invoking the expert persona reduced lay agreement. This suggests that current LLMs, even with expert prompting, primarily reflect general-population (RLHF) tuning rather than capturing domain-specific expertise (Szymanski et al., 26 Oct 2024).

4. Identified Limitations of LLM-Only Juries in Expert Domains

Key constraints of an LLM-only jury are empirically surfaced:

• Surface-level pattern-matching: LLM judges often overvalue superficial textual criteria (length, keyword overlap) and under-detect deeper factual errors or risks highlighted by SMEs.
• Lack of domain nuance: LLM rationales tend to restate prompt content or paraphrase the candidate outputs, rarely reflecting context-dependent trade-offs or the tacit knowledge used by human experts.
• Risk insensitivity: Especially in sensitive fields (e.g., mental health), LLMs are less likely to detect outputs that, while factually correct, entail serious practical risks (e.g., inducing anxiety).
• RLHF-induced lay bias: Elevated lay-LLM agreement, compared to SME-LLM, suggests that LLMs are more reflective of general-population preferences rather than true expert criteria.
• Expert persona limitations: Adopting an expert persona sometimes increases alignment for certain aspects (accuracy, standards) but can decrease it elsewhere (clarity, educational context), underscoring the inadequacy of superficial persona prompting absent domain-tuned training.

The implication is that sole reliance on LLMs for evaluation in specialist domains will systematically fail to capture expert-level safety and contextual reasoning requirements (Szymanski et al., 26 Oct 2024).

5. Workflow and Best-Practice Guidelines for LLM-Jury Deployment

A robust LLM-Jury pipeline requires a hybrid, modular architecture:

1. SME-in-the-loop evaluation: Use LLM juries for broad, automated screening; retain final validation and high-stakes rejection authority for SMEs, particularly in cases flagged for disagreement or potential harm.
2. Domain-specific evaluation design: Construct challenging, realistic prompt sets and tailored aspect-questions for each professional field, ensuring both LLMs and SMEs are evaluated under identical conditions.
3. Persona priming plus fine-tuning: While prompt-level expert personas can partially boost domain alignment, true performance improvement is only achieved by fine-tuning or reward modeling LLM judges using a small, high-quality set of SME-labeled judgments.
4. Multi-metric reporting: Always report both the raw agreement and reliability-adjusted metrics (percent agreement, Cohen’s κ, confidence intervals). When scalar scores are used, report correlation coefficients.
5. Thematic, explanation-based auditing: Periodically code and analyze free-text LLM rationales against SME-labeled themes (e.g., clinical accuracy, risk-mitigation) to detect persistent blind spots.
6. Ongoing feedback and reevaluation: As expert standards evolve, periodically refresh prompt sets, SME criteria, and LLM fine-tuning data to maintain alignment (Szymanski et al., 26 Oct 2024).

This pipeline aims to combine the throughput and cost efficiency of LLM screening with the necessary rigor and risk-mitigation of expert oversight.

6. Broader Implications: Domain Portability and Research Significance

Empirical findings in high-stakes domains suggest that while large-scale LLM-jury workflows enable scalable, transparent, and reproducible measurement of model performance, they must be carefully designed to avoid favoring general-population proxies over true professional standards. As models and fine-tuning recipes evolve, periodic SME–LLM consensus tracking should be institutionalized.

Researchers are encouraged to use percent agreement, Cohen’s κ, rationale coding, and cross-domain comparisons in reporting, making explicit both the efficiency gains and the safety limitations of the jury approach. When evaluating the deployment of LLM-Jury in novel domains, bespoke aspect-pooled test sets calibrated against subject-matter input remain essential.

The LLM-Jury methodology provides a rigorously tested, readily extensible template for future multi-level, domain-specific, and safety-critical model evaluation efforts. Its deployment supports the scalable oversight of LLM ecosystems, but only when paired with cyclical, domain-aware SME involvement and transparent, reproducible metric design (Szymanski et al., 26 Oct 2024).