---
title: Multi-Agent Judging Framework
url: https://www.emergentmind.com/topics/multi-agent-judging-framework
type: topic
---

# Multi-Agent Judging Framework

A Multi-Agent Judging Framework comprises a set of autonomous agents—each typically instantiated as a Small Language Model (SLM) or Large Language Model (LLM) and assigned distinct roles—that collectively evaluate responses to adversarial prompts or other tasks through structured debate and consensus-building. This paradigm is particularly prominent in scalable safety assessment of LLMs, where cost reduction and semantic fidelity are crucial. The approach leverages adversarial role conditioning, inter-agent debate, and formal aggregation rules to capture nuanced violations and achieve reliability comparable to high-cost frontier models, but at an order-of-magnitude lower inference cost.

## 1. Architecture: Critic, Defender, and Judge Agents

The framework [2511.06396] instantiates three core agents:
- **Critic Agent (C):** Given a prompt–response pair $(x, y)$, the Critic scores the response across $K = 5$ fixed safety aspects (e.g., toxicity, privacy, illegal advice), producing for each aspect $a_k$ a sub-score $u_k \in [1,10]$, a natural-language critique, an aggregated risk level (1–5), and an overall ten-point score.
- **Defender Agent (D):** Receives Critic outputs $(u_k, m_k)$ and for each aspect rebutts with counter-score $v_k \in [1,10]$ and a defense explanation, aiming to lower perceived risk and simulate adversarial robustness.
- **Judge Agent (J):** After $R$ rounds of Critic–Defender interaction, the Judge integrates all arguments and outputs final attack success (binary), five-level risk, and ten-point continuous risk, with a rationale.

Agent specialization is achieved by prompt engineering and context conditioning; agents share the same base model, but system-level prompts encode distinct roles and evaluation rubrics.

## 2. Debate Protocol and Value Alignment

Each evaluation proceeds through structured rounds:
- **Pre-Debate Alignment:** All agents are initialized with a shared, explicit taxonomy of $K$ safety aspects to constrain scope and prevent topic drift.
- **Round Structure:** In each round $r$, Critic first produces per-aspect scores and messages; Defender rebuts with counter-scores and responses. This is strictly turn-taking; agents do not directly alter one another's scores from previous rounds.
- **Stopping Criterion:** $R = 3$ rounds empirically yields optimal accuracy–cost tradeoff; further rounds introduce error accumulation and diminish returns. Optionally, early exit occurs if scores stabilize without substantive changes.

Three rounds are found to be the “sweet spot”: maximal improvement in semantic reconciliation with token cost growing linearly in $R$.

## 3. Formal Scoring and Aggregation Mechanics

Judging is formalized by aggregation equations:
- **Critic per-aspect utility:**
  \[
  u_{r, k} = U_c(x, y; a_k), \quad U_c: (x, y, a) \mapsto [1,10]
  \]
- **Aggregate per round:**
  \[
  U_c^r(x, y) = \frac{1}{K}\sum_{k=1}^K u_{r, k}
  \]
- **Defender counter-argument:**
  \[
  v_{r, k} = S_d(x, y, u_{r, k}, a_k), \quad S_d: (x, y, u, a) \mapsto [1,10]
  \]
- **Defender aggregate:**
  \[
  U_d^r(x, y) = \frac{1}{K}\sum_{k=1}^K v_{r, k}
  \]
- **Judge decision rule:**
  \[
  D_j(c_1, d_1, \ldots, c_R, d_R) = \lambda \frac{1}{R} \sum_{r=1}^R U_c^r + (1-\lambda) \frac{1}{R} \sum_{r=1}^R U_d^r
  \]
Continuous final risk score $D_j \in [1, 10]$; attack success is declared if $D_j > \tau$ (threshold $\tau$ selected by maximizing Cohen's $\kappa$ on validation). Mixing weight $\lambda$ and threshold $\tau$ are tuned for maximum ground-truth agreement.

## 4. Calibration, Prompt Engineering, and Training

No gradient-based fine-tuning is applied. Calibration relies on:
- **Prompt Engineering:** All agents utilize rigorously engineered prompts encoding evaluation taxonomy and roles.
- **Pre-Debate Alignment:** Ensures agents evaluate on the same $K$ aspects.
- **Noise Filter:** Optional pre-processing to eliminate adversarial artifacts from inputs with additional LLM-based filtering.
- **Threshold/Weight Tuning:** Only $\tau$ (success boundary) and $\lambda$ (critic–defender mixing) are learned, aligning agent consensus with HAJailBench human labels.

## 5. Inference Cost Reduction and Resource Efficiency

Each inference comprises $2R + 1$ SLM calls ($R=3$ implies 7 calls); SLMs such as Qwen3-14B incur $\sim$5x less unit cost than GPT-4o judges. Overall, the multi-agent protocol requires only $0.46\times$ the cost of a single GPT-4o call, even when accounting for extra rounds. Lower per-call context size and parallelization further enhance token efficiency.

## 6. Empirical Evaluation: HAJailBench Benchmark

- **Dataset:** HAJailBench collects 12,000 adversarial instances (100 harmful goals, 12 attack methods, multiple target LLMs) with expert-annotated ground truth (binary success/fail, 5-level risk, 10-point risk).
- **Metrics:** Cohen’s $\kappa$ (agreement with ground truth), unit cost/query, cost ratio with GPT-4o baseline.
- **Results:**
  - GPT-4o judge: $\kappa=0.7627$, cost = \$8.36×10⁻⁴/query
  - Qwen3-14B multi-agent: $\kappa=0.7352$, cost = \$3.85×10⁻⁴/query (54% lower)
  - Multi-agent improves over single SLM JailJudge by 25–32% relative $\kappa$ gain, reduces cost by 54–82%.

### Debate Round Ablation

| Rounds | $\kappa$   | Cost Ratio vs Optimal (R=3) |
|--------|-----------|-----------------------------|
| 0      | 0.5709    | 0.30                        |
| 1      | 0.6955    | 0.69                        |
| 2      | 0.7143    | 0.87                        |
| 3      | 0.7352    | 1.00 (optimal)              |
| 4      | 0.7260    | 1.05                        |
| 5      | 0.7221    | 1.12                        |

Accuracy sharply increases up to three rounds; further rounds introduce error drift, plateau, or degrade consensus.

## 7. Insights, Limitations, and Extensions

- **Key Insights:** Structured debate with value-aligned roles enables cost-effective SLM ensembles to match semantic richness and reliability of large judge models for safety evaluation; fixed safety aspects anchor reasoning; maximal benefit realized after three rounds.
- **Limitations:** HAJailBench covers 100 goals × 12 attacks; unseen or emergent attacks may challenge reliability. SLM-based agents are susceptible to multi-turn hallucinations and bias stacking. Human annotation in HAJailBench references a large-model judge in the second round, potentially introducing bias.
- **Potential Extensions:** Dynamic topic reallocation, uncertainty-aware adaptive stopping, adversarial robustness mechanisms in debaters, human-in-the-loop review, continual learning, and cross-cultural calibration are identified as promising future directions.

---

A Multi-Agent Judging Framework, as exemplified by the Critic–Defender–Judge protocol [2511.06396], organizes refinement and consensus-building in safety evaluation via structured, prompt-conditioned SLM agents, with provable gains in semantic fidelity and cost efficiency on large-scale adversarial datasets. This modular debate architecture is effective in capturing and aligning nuanced jailbreak risks, setting a precedent for scalable, interpretable LLM safety assessment.

Source: https://www.emergentmind.com/topics/multi-agent-judging-framework