AggreFact: Factuality Benchmark
- AggreFact is a benchmark that aggregates factuality datasets to evaluate summary-level faithfulness and document-grounded claim verification.
- It organizes data into binary classification tasks using metrics like balanced accuracy and F1, addressing challenges such as class imbalance and evidence synthesis.
- The benchmark supports both summarization-specific and broader LLM‑AggreFact evaluations, serving as a diagnostic tool for various factuality detection methods.
Searching arXiv for papers directly relevant to AggreFact and its use as a factuality benchmark. AggreFact is a benchmark family for evaluating factuality or faithfulness in document-grounded generation, especially summarization, and it is used in later work both in its original summarization-oriented form and in the broader claim-verification aggregation LLM‑AggreFact. In the summarization setting, AggreFact is treated as a document–summary binary faithfulness classification task with labels such as “faithful” vs “unfaithful” at the summary level for CNN/DailyMail and XSum (Onyshchuk et al., 11 Jun 2026). In later evaluator work, LLM‑AggreFact is described as “a combination of 10 datasets covering three tasks: fact verification, summarization, and long-form QA,” with all datasets expressed as human-annotated (document, claim, label) tuples, thereby extending the AggreFact paradigm from summary-level faithfulness to a more general document-grounded claim-verification benchmark (Xie et al., 2024). Across these uses, AggreFact functions less as a single dataset than as a standardized evaluation substrate for testing whether detectors, judges, or internal probes can distinguish supported from unsupported generated content.
1. Origins, scope, and benchmark variants
AggreFact is explicitly described as a benchmark that aggregates prior factuality resources for summarization. One account characterizes AGGREFACT as “the de-facto benchmark for evaluating summarization factuality metrics,” with manually annotated summaries labeled “factual” or “non-factual,” and with splits corresponding to FTSOTA, ExFORMER, and OLD system generations over CNN/DailyMail and XSum (Scirè et al., 2024). Another account states that AggreFact “aggregates factuality annotations from nine existing datasets for summarization” and is stratified into FtSota, EXformer, and Old, with the paper focusing on AggreFact‑FtSota because it “most accurately reflects the challenges of unfaithfulness in the most advanced summarization models” (Qiu et al., 2023).
A later line of work broadens this notion into LLM‑AggreFact. In that setting, the benchmark aggregates multiple document-grounded verification tasks rather than only summarization, and it is used to evaluate factuality judges on a shared binary or collapsed binary label space (Xie et al., 2024). Another paper describes LLM‑AggreFact as aggregating 11 factuality datasets, with task columns including AggreFact (CNN, XSum), TofuEval (MediaS, MeetB), WiCE, REVEAL, ClaimVerify, FactCheck, ExpertQA, LFQA, and RAGTruth (Lin et al., 6 Jun 2026). A reasoning-trace study similarly frames LLMAggreFact as aggregating nine claim-verification datasets spanning news, science, dialogue, and other domains, and analyzes them as a common testbed for reasoning patterns in claim verification (Rao et al., 2 Apr 2026).
This suggests a two-level terminology. The narrower form, AggreFact, denotes the summarization factuality benchmark centered on CNN/DailyMail and XSum. The broader form, LLM‑AggreFact, denotes an aggregated document–claim verification benchmark that includes AggreFact as one component and is intended for evaluating general-purpose factuality judges and detectors (Xie et al., 2024).
2. Data organization and task formulations
In summarization-focused work, AggreFact is used in a collapsed binary format. One study reports using the AggreFact test splits for CNN/DailyMail: and XSum: , for a total of examples, and treats the dataset as a document–summary binary faithfulness classification task in which labels are used “only for evaluation,” not for training the detector (Onyshchuk et al., 11 Jun 2026). The same source notes that CNN/DailyMail is heavily skewed toward faithful summaries (about faithful) and therefore emphasizes balanced accuracy as the primary metric (Onyshchuk et al., 11 Jun 2026).
For the FtSota splits used in factuality-metric evaluation, one paper gives exact validation and test counts. AggreFact‑Cnn‑FtSota contains validation subsets from Polytope 34, SummEval 200, FRANK 75, CLIFF 150, totaling 459, and test subsets Polytope 34, SummEval 200, FRANK 175, CLIFF 150, totaling 559. AggreFact‑Xsum‑FtSota contains validation subsets Wang’20 120, CLIFF 150, Goyal’21 50, Cao’22 457, totaling 777, and test subsets Wang’20 119, CLIFF 150, Goyal’21 50, Cao’22 239, totaling 558 (Qiu et al., 2023). This decomposition underscores that AggreFact is an aggregation of several preexisting factuality resources rather than a monolithic collection.
In evaluator work, the task is reformulated at the claim level. LLM‑AggreFact is described as “a combination of 10 datasets covering three tasks: fact verification, summarization, and long-form QA,” where all examples are (document, claim, label) tuples (Xie et al., 2024). The ten component datasets are listed as AggreFact (CNN, XSum), TofuEval (MediaS, MeetB), Wice, Reveal, Claim Verify, Fact Check, Expert QA, and Lfqa (Xie et al., 2024). A separate small-reasoning-model paper lists an overlapping but slightly expanded composition, explicitly including RAGTruth among the components used for evaluation on LLM‑AggreFact (Bergeron et al., 1 Oct 2025).
The document-grounded label semantics are also standardized. One paper defines the binary setting as supported/consistent vs contradicted/inconsistent for a document–claim pair, with the detector output interpreted as the probability that the claim is consistent with the document (Lin et al., 6 Jun 2026). Another describes the task as classifying a claim as grounded or hallucinated, where intrinsic and extrinsic hallucinations are collapsed into a single hallucinated label for evaluation (Bergeron et al., 1 Oct 2025).
3. Evaluation protocols and dominant metrics
AggreFact is closely associated with balanced accuracy. In the summarization transfer study, balanced accuracy is explicitly defined as
and is used because of “strong class imbalance, especially on CNN/DailyMail ( of summaries are faithful)” (Onyshchuk et al., 11 Jun 2026). The same paper also reports a random baseline of 50.0% BAcc for AggreFact experiments (Onyshchuk et al., 11 Jun 2026).
The factuality-metric literature uses the same metric. FENICE evaluates summary-level scores by thresholding them into binary predictions and measuring balanced accuracy, with thresholds tuned on validation splits and then applied to test splits (Scirè et al., 2024). For FTSOTA, it also reports a single-threshold setting in which one threshold is tuned jointly on the combined validation sets of CNN‑FTSOTA and XSum‑FTSOTA and then applied to the combined test set (Scirè et al., 2024). AMRFact likewise reports balanced binary accuracy (%) on AggreFact‑Cnn‑FtSota and AggreFact‑Xsum‑FtSota, including confidence intervals (Qiu et al., 2023).
In LLM‑AggreFact, later work often reports F1 instead. CCHD states that evaluation “follows the LLM‑AggreFact benchmark” and reports per-task F1 as well as macro-averaged AVG across 11 factuality tasks (Lin et al., 6 Jun 2026). CPIL uses the same setup, reporting per-task and macro F1 on the 11-task LLM‑AggreFact benchmark (Lin et al., 6 Jun 2026). By contrast, evaluator work such as FenCE reports balanced accuracy (BAcc) on the LLM‑AggreFact test split, both overall and per component dataset (Xie et al., 2024). This metric variation reflects benchmark reuse across somewhat different communities: summary-level metric evaluation has favored balanced accuracy, whereas detector-training papers on LLM‑AggreFact often favor macro F1.
Several papers also emphasize that evaluation settings matter as much as the metric itself. DEEP argues that many factuality metrics perform much worse when thresholds are not optimized on subsets of the evaluated dataset, and motivates a leave-one-dataset-out setup precisely because target-domain threshold tuning is “unrealistic in practice” (Chandler et al., 2024). This suggests that part of AggreFact’s significance lies in the evaluation protocol it induces: metrics are judged not only by raw scoring quality but by how well they transfer across strata, domains, and summarizer regimes.
4. AggreFact in summarization factuality research
AggreFact has become a central benchmark for summary-level factuality metrics. FENICE describes it as the de-facto benchmark for summarization factuality evaluation and reports a new state of the art on AGGREFACT by combining claim extraction with NLI-based alignment, coreference-aware refinement, and multi-granularity context matching (Scirè et al., 2024). On the six-way AGGREFACT split comprising Agg-CNN FTS/EXF/OLD and Agg-XSum FTS/EXF/OLD, FENICE achieves an AVG balanced accuracy of 74.0, while FENICE reaches 72.3 (Scirè et al., 2024).
AMRFact focuses on AggreFact‑FtSota, especially AggreFact‑Xsum‑FtSota and AggreFact‑Cnn‑FtSota, and argues that this split best reflects factuality challenges in modern transformer summarizers (Qiu et al., 2023). It reports balanced accuracy of 72.3 ± 2.5 on AggreFact‑Cnn‑FtSota, 64.1 ± 1.8 on AggreFact‑Xsum‑FtSota, and 68.2 average, outperforming prior non-LLM baselines and rivaling GPT‑4-based evaluation (Qiu et al., 2023). Its analysis further shows that removing any of its five AMR-driven perturbation types harms performance on AggreFact‑FtSota, with the largest drop occurring when discourse link errors are removed (Qiu et al., 2023).
DEEP uses AggreFact‑XSUM FTSOTA as one of its main benchmarks, excluding the CNN/DM counterpart because of “its insufficient number of factual inconsistencies and its more extractive summary style” (Chandler et al., 2024). Under a cross-dataset setting with no target-specific threshold tuning, its best ensemble achieves 71.9% balanced accuracy on AggreFact‑XSUM FTSOTA (Chandler et al., 2024). DEEP’s analysis is especially notable because it stresses that many prior factuality metrics degrade sharply when thresholds are transferred rather than tuned on the target AggreFact subset (Chandler et al., 2024).
The recurring role of AggreFact here is diagnostic. It is used not merely to establish leaderboard orderings, but to stress-test specific modeling assumptions: claim decomposition in FENICE, AMR-structured perturbation in AMRFact, and prompt ensembling plus calibration in DEEP.
5. AggreFact as a testbed for unsupervised, supervised, and label-efficient detectors
AggreFact also serves as a benchmark for detectors that are not specific to summarization metrics. In a study of optimal-transport signals in attention, AggreFact is the main benchmark used to test whether a geometric hallucination signal learned from NMT transfers to summarization (Onyshchuk et al., 11 Jun 2026). There the unsupervised OT Detector (T5-base) achieves 57.2% balanced accuracy on CNN, 57.6% on XSum, and 57.4% average; OT Detector (Flan‑T5‑large) achieves 55.6% on CNN, 61.4% on XSum, and 58.5% average; and MiniCheck‑Flan‑T5‑L reaches 69.9% on CNN, 74.3% on XSum, and 72.1% average (Onyshchuk et al., 11 Jun 2026). The same paper interprets the gap as principled: unlike NMT hallucinations, unfaithful summaries can attend to the correct source content while misrepresenting it, making cross-attention concentration metrics fundamentally limited on AggreFact (Onyshchuk et al., 11 Jun 2026).
At the broader LLM‑AggreFact level, FenCE uses the benchmark as its primary external testbed for claim-level factuality evaluation (Xie et al., 2024). It reports average BAcc of 66.4 for base Llama3‑8B‑chat, 71.8 for FenCE (Vanilla SFT), 73.7 for FenCE (Critique Only), and 74.7 for FenCE (Full) (Xie et al., 2024). It further reports per-dataset scores including AggreFact (CNN): 62.1 and AggreFact (XSum): 72.4 for FenCE (Full) (Xie et al., 2024). The benchmark’s role here is explicitly out-of-distribution evaluation: FenCE is not trained on LLM‑AggreFact but validated on it as a general-purpose factuality judge (Xie et al., 2024).
CCHD and CPIL use LLM‑AggreFact to demonstrate improvements in document–claim hallucination detection. CCHD reports macro AVG F1 of 79.73 for CCHD‑DBT and 79.11 for CCHD‑FT5, outperforming FactCG, MiniCheck, and AlignScore, with especially strong gains on AggreFact‑CNN and AggreFact‑XSum (Lin et al., 6 Jun 2026). CPIL, with only about 1% labeled data, achieves macro AVG F1 of 79.14 on the 11-task LLM‑AggreFact benchmark, surpassing FactCG at 77.14, MiniCheck at 76.47, and AlignScore at 74.23 (Lin et al., 6 Jun 2026). These studies position AggreFact not just as a benchmark for metric scoring, but as a standardized supervised learning target for detector design.
6. Benchmark analysis, reasoning coverage, and limitations
AggreFact’s importance has also made it the subject of benchmark critique. A reasoning-trace analysis asks what claim-verification datasets, including LLMAggreFact, actually test (Rao et al., 2 Apr 2026). Using structured reasoning traces on 24.1K examples across 9 datasets, the study finds that direct evidence extraction / lexical matching dominates, while multi-sentence synthesis and numerical reasoning are under-represented (Rao et al., 2 Apr 2026). It reports that AggreFact‑CNN “requires information synthesis in roughly half of cases,” making it relatively harder than strongly lexical datasets, while FactCheck‑GPT and Wice show more prominence of scope, nuance, and absence-of-evidence patterns (Rao et al., 2 Apr 2026).
The same study argues that high benchmark scores on AggreFact-like resources primarily reflect retrieval-plus-entailment ability rather than deep reasoning (Rao et al., 2 Apr 2026). Error analyses with a compact reasoning verifier identify lexical overlap bias, insufficient aggregation, negation / temporal confusion, overcautiousness, and hallucinated justification, with general-domain verification dominated by lexical overlap bias (Rao et al., 2 Apr 2026). This directly tempers interpretations of strong LLM‑AggreFact performance: such performance may certify competence at local evidence matching more than at causal, temporal, or numerical reasoning.
A different limitation emerges in summarization transfer work. On AggreFact, attention-based OT metrics achieve only modest performance because many unfaithful summaries involve content misuse rather than source disengagement (Onyshchuk et al., 11 Jun 2026). The paper states that “unfaithful summaries can attend correctly to source tokens while misrepresenting their content,” making that failure mode “invisible to concentration-based OT metrics by construction” (Onyshchuk et al., 11 Jun 2026). This suggests that AggreFact captures factuality errors that occur downstream of retrieval, and therefore exposes the insufficiency of detectors that only inspect where attention goes.
Another limitation is annotation or framing mismatch. AMRFact notes that a portion of its apparent errors on AggreFact‑FtSota are attributable to annotation errors, and also remarks that some “out-of-article” statements may be true according to world knowledge while still being labeled unfaithful relative to the document (Qiu et al., 2023). A plausible implication is that AggreFact operationalizes document-grounded faithfulness, not world-truth in a broader epistemic sense.
7. Relation to modern evaluator and detector design
AggreFact’s continued use in 2024–2026 work shows that it has become a common substrate for several distinct methodological agendas. In FLAMe, LLM‑AggreFact is one of the main held-out autorater benchmarks, and FLAMe‑24B achieves 81.1 overall, slightly above GPT‑4‑0125 at 80.6 and GPT‑4o at 80.2 (Vu et al., 2024). The same paper decomposes LLM‑AggreFact into LLM‑FactVerify, Wiki‑FactVerify, Summarization, and Long‑form QA, with FLAMe variants outperforming other models in three of the four categories (Vu et al., 2024). This positions AggreFact as a benchmark not only for factuality metrics, but also for foundational autoraters trained on diverse human-judgment mixtures.
In RAG and internal-representation work, AggreFact functions as an external reference-based factuality benchmark to test generalization beyond the training domain. RAGLens uses AggreFact as one of its external datasets for hallucination detection and reports cross-dataset AUC of 0.8019 when trained on RAGTruth and tested on AggreFact, compared to 0.5741 for chain-of-thought self-judgment (Xiong et al., 9 Dec 2025). The same paper also reports that training directly on AggreFact yields AUC ~0.83 on AggreFact, and argues that sparse autoencoder features can complement AggreFact-style external evaluation by providing internal, token-level rationales (Xiong et al., 9 Dec 2025).
HalluGuard uses the full LLM‑AggreFact benchmark purely for evaluation and reports 75.7% average balanced accuracy, with 84.0% on the RAGTruth subset, matching MiniCheck‑7B on RAGTruth while using a 4B small reasoning model (Bergeron et al., 1 Oct 2025). Here AggreFact functions as a unifying benchmark for document–claim grounding across summarization, RAG, QA, and fact verification, and is used to validate that a small model can rival much larger judges (Bergeron et al., 1 Oct 2025).
These uses indicate that AggreFact has become a boundary object across subfields. Summarization researchers use it to evaluate faithfulness metrics. Hallucination-detection researchers use it to compare supervised and unsupervised detectors. Evaluator researchers use its aggregated form to test judge generalization. Interpretability researchers use it to anchor internal probes against an external factuality benchmark.
8. Benchmark significance and continuing research directions
AggreFact’s significance lies in its dual standardization. First, it standardizes task framing: document-grounded factuality is cast as either summary-level faithfulness or claim-level support classification. Second, it standardizes evaluation culture: systems are compared across shared subsets, with attention to model era, dataset difficulty, and label imbalance.
The benchmark has also shaped methodological recommendations. Work on OT-based detection concludes that for AggreFact and similar summarization benchmarks, one should not rely solely on attention-based concentration or geometry as a faithfulness signal, and should instead combine such signals with semantic reasoning or NLI-style detectors (Onyshchuk et al., 11 Jun 2026). FenCE suggests that strong LLM‑AggreFact performance can serve as the foundation for better downstream training signals in generator factuality optimization (Xie et al., 2024). CCHD suggests that paraphrase-consistency constraints can improve hallucination detection on AggreFact-style tasks without adding inference-time cost (Lin et al., 6 Jun 2026). CPIL suggests that cross-paraphrastic invariance and same-document hard negatives are particularly effective under label scarcity on the LLM‑AggreFact benchmark (Lin et al., 6 Jun 2026). The reasoning-trace analysis recommends adding more multi-hop, numerical, and partial-support cases to AggreFact-like benchmarks to better test real reasoning rather than shallow lexical matching (Rao et al., 2 Apr 2026).
A plausible implication is that AggreFact now serves two roles at once: it is both a benchmark to beat and a probe revealing what current factuality systems can and cannot do. High performance on AggreFact is widely treated as meaningful, but several papers also show that such performance does not settle deeper questions about semantic transformation, numerical reasoning, or causal inference (Rao et al., 2 Apr 2026). This tension helps explain why AggreFact remains central: it is sufficiently established to anchor comparison, yet sufficiently challenging and heterogeneous to continue exposing failure modes.