---
title: 'MAD-Fact: Multi-Agent Factuality Check'
url: https://www.emergentmind.com/topics/mad-fact
type: topic
---

# MAD-Fact: Multi-Agent Factuality Check

MAD-Fact refers to a debate-based, multi-agent factuality evaluation framework designed for long-form outputs from large language models (LLMs). It integrates structured claim decomposition, heterogeneous agent deliberation, and a fact importance hierarchy, supported by a new benchmark, LongHalluQA. MAD-Fact advances the assessment of factual reliability in complex, multi-claim generative texts, targeting safe LLM deployment in high-risk or regulated domains [2510.22967].

## 1. Motivation and Problem Setting

LLMs routinely generate long-form answers whose factual accuracy is crucial in domains such as medicine, law, and education. Existing evaluation tools for short texts (e.g., FactScore, FacTool, Q^2) struggle with reasoning chains, cumulative errors, and the non-uniform importance of individual claims found in long responses. Moreover, subjective, single-model annotation may miss subtle inconsistencies and be vulnerable to systematic bias.

MAD-Fact addresses these limitations by:
- Decomposing long-form answers into atomic claims.
- Engaging multiple agents with diverse expertise and retrieval strategies in a debate over each claim.
- Aggregating consensus verdicts using a weighted scheme that emphasizes higher-importance facts [2510.22967].

## 2. System Architecture and Protocol

MAD-Fact implements a three-stage architecture:

### (i) Clerk (Claim Extraction)
Receives $(q_i, a_i)$, where $q_i$ is a question and $a_i$ is a long-form LLM-generated answer. It outputs a set of $T$ atomic factual claims $\{c_{i,1}, \dots, c_{i,T}\}$. Formally,
\[
\{c_{i,j}\}_{j=1}^T = \mathrm{Clerk}(conf_\mathrm{Clerk}, q_i, a_i)
\]

### (ii) Jury of Evaluators (Multi-Agent Debate)
A group of $N$ role-specialized evaluator agents $\mathrm{Evaluator}^n$, each simulating a different perspective (General Public, Critic, Scientist, Data Analyst, Psychologist, News Author), debate each extracted claim $c_{i,j}$. Debate proceeds for up to two rounds per claim, with each agent studying previous statements and optionally retrieving fresh evidence. For each round and claim, the acting agent returns:
- A direct judgement $p_{i,j,t}\in\{\mathrm{TRUE}, \mathrm{FALSE}\}$
- An explanation $e_{i,j,t}$
- Optionally, a retrieval query with externally sourced $k_{i,j,t}$

At the end, the full message pool $M_{i,j,t}$ (round-by-round debate log) and knowledge pool $K_{i,j,t}$ (retrieved evidences) form the factuality context for each claim.

### (iii) Judge (Aggregation and Verdict)
A judge agent aggregates the $N$ final votes for each claim, $p^n_{i,j}$, via majority rule ($\mathrm{Judge}(\cdot)$), breaking ties by giving priority to the last speaker. This produces binary verdicts $\{p_{i,j}\}_{j=1}^T$ for all claims. An overall factuality score is computed over all claims, weighted by their hierarchical importance (see below).

## 3. Fact Importance Hierarchy and Weighted Metrics

Recognizing that some claims in a long answer are more critical than others, MAD-Fact introduces a fact importance hierarchy:

- Reference answers for a given question are generated by $G$ expert models (e.g., GPT-4, Claude-3, DeepSeek).
- Each reference is decomposed into atomic claims.
- The cross-expert merger aligns semantically equivalent claims, forming a "golden set" $\{g_{i,k}\}_{k=1}^{K_{\mathrm{gold}}}$.
- Claim frequency across experts $f(g_{i,k})$ defines its importance; the fewer experts mentioning a fact, the less critical its content.
- Each claim receives a level $\ell(g_{i,k})=G-f(g_{i,k})+1$; pre-set monotonic weights $\omega_1 > \omega_2 > \dots > \omega_G$ are assigned, e.g., $\omega_\ell=5-\ell$.

The answer-level precision, recall, and F1 are then reweighted accordingly:
\[
\mathrm{Prec}_w(a_i) =\frac{\sum_{j:p_{i,j}=\mathrm{TRUE}}\omega_{i,j}}{\sum_{j=1}^{T}\omega_{i,j}}
\qquad
R_w@\gamma(a_i)=\min\Bigl(1, \frac1\gamma \frac{\sum_{j:p_{i,j}=\mathrm{TRUE}}\omega_{i,j}}{\sum_{k=1}^{K_{\mathrm{gold}}}\omega_{i,k}}\Bigr)
\]
with $F_1@\gamma$ given as the harmonic mean, and a recall "slack" parameter $\gamma \le 1$ to tolerate length/coverage mismatches [2510.22967].

## 4. The LongHalluQA Dataset

LongHalluQA is a large-scale Chinese long-form factuality benchmark, constructed to evaluate multi-agent debate frameworks:

- Contains 2,746 samples from seven domains (Chinese culture, natural sciences, social sciences, ...)
- Average answer length is over 9 times that of original HalluQA
- Each sample includes reference knowledge, sub-question expansion, and manual verification of factual consistency and clarity.

This dataset enables robust quantitative and qualitative evaluation of factual verification protocols under realistic, high-information-load conditions.

## 5. Experimental Results

MAD-Fact was evaluated on both standard fact-checking tasks and long-form outputs. The core pipeline employs GPT-4o-mini for the Clerk and Evaluators, with Llama3.3-70B-instr as Judge.

### Fact-Checking Benchmark Performance

Across tasks such as Factcheck-Bench, FacToolQA, BingCheck, FELM-WK, and FactEval-CN:

| Method    | TRUE F1 (range) | FALSE F1 (range) |
|-----------|-----------------|------------------|
| SAFE      | 0.68–0.87       | 0.51–0.74        |
| FIRE      | 0.71–0.88       | 0.53–0.70        |
| MAD-Fact  | 0.74–0.88       | 0.64–0.73        |

MAD-Fact outperforms strong baselines in 8/10 F1 comparisons and yields improvements in both factual error sensitivity and correct claim recall.

### Long-Form Scoring

Weighted $F_1@\gamma$ (with $\gamma=1.0, 0.8$) on held-out LongFact and LongHalluQA splits:

- On LongFact, GPT-4-Turbo achieves $F_1@0.8=0.681$, Doubao-1.5-Pro 0.669, Qwen2.5-72B 0.648.
- On LongHalluQA, Chinese models (QwQ-32B, Doubao-1.5-Pro) approach or exceed GPT-4 by up to 10 points, likely due to in-domain pretraining advantages.

## 6. Design Insights and Case Analysis

- **Agent diversity**: Multiple conversational roles, retrieval strategies (autonomous, mandatory, adaptive), and external evidence sources contribute to more stable and error-corrective debate outcomes.
- **Importance reweighting**: Emphasizing pivotal claims leads to higher-quality, actionable factuality scores, addressing the unequal significance of content.
- **Consensus stabilization**: Peer agents collectively correct judgment errors and prevent overconfident misclassification by promoting deliberative discussion.
- **Robustness**: The combination of single-model, dual-model, and jury voting enables performance that generalizes well even on unseen, long-form user queries.

## 7. Limitations and Future Directions

- **Dataset scope**: LongHalluQA is built from trusted knowledge bases, not live user logs. Further research should extend to dynamic, domain-specific, or real-world user-generated queries.
- **Scalability**: MAD-Fact increases inference cost due to agent multiplicity and external retrieval. Dynamic early termination, adaptive agent allocation, and parameter-efficient agent architectures are key optimization areas.
- **Residual bias and groupthink**: Multi-agent debates can still become overconfident in converging on spurious claims. Future work includes confidence-weighted voting, formal cross-agent logical checks, and adversarial stress-testing with planted falsehoods.

MAD-Fact provides a transparent, interpretable, and scalable framework for nuanced long-form factuality assessment, addressing gaps in both claim prioritization and adversarial model collaboration [2510.22967]. Its dataset and architecture serve as baselines for advancing multi-perspective, retrieval-augmented LLM evaluation protocols in critical safety contexts.

Source: https://www.emergentmind.com/topics/mad-fact