DeepFact-Bench: DRR Factuality Benchmark

• The paper introduces DeepFact-Bench, a benchmark for claim-level factuality in DRRs that evolves through iterative Audit-then-Score rounds.
• It employs a dynamic auditing process where experts and models resolve label disagreements with revisable, evidence-backed rationales.
• Empirical results demonstrate expert accuracy improvements from 60.8% to 90.9% over four rounds, enhancing reliability in factuality evaluation.

Searching arXiv for DeepFact-Bench and closely related factuality benchmarking work. DeepFact-Bench is a versioned benchmark for claim-level factuality in deep research reports (DRRs), instantiated through an evolving evaluation paradigm called Audit-then-Score (AtS). It is designed for a setting in which search-augmented LLM agents can produce DRRs, yet verifying factuality at the level of individual claims remains difficult and poorly served by benchmarks built for general-domain, factoid-style atomic claims. Its defining property is that labels and rationales are auditable and explicitly revisable: when a verifier disagrees with the current benchmark, it can challenge the label with evidence, an auditor adjudicates the dispute, and accepted revisions update the benchmark before scoring. In the reported construction process, expert accuracy on a hidden micro-gold set increased from 60.8% to 90.9% across four AtS rounds, motivating DeepFact-Bench as a co-evolving alternative to static expert-labeled factuality datasets (Huang et al., 6 Mar 2026).

1. Problem setting and motivation

DeepFact-Bench addresses factuality verification for DRRs, a regime in which claims are embedded in long, domain-specialized documents and often require substantial context, expertise, and sustained attention to verify. The motivating observation is that existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims, and there was no benchmark to test whether such verifiers transfer to DRRs (Huang et al., 6 Mar 2026).

The benchmark is motivated by a critique of static expert-labeled evaluation. In a controlled study with PhD-level specialists, unassisted experts achieved only 60.8% accuracy on a hidden micro-gold set of verifiable claims in their own specialty domains. The reported interpretation is that static, one-shot expert-labeled datasets are unreliable for high-level factuality verification in complex, document-level settings. The paper further attributes this brittleness to expert fallibility, annotation noise, and the practical difficulty of scaling redundancy or multi-expert adjudication when verification is cognitively demanding (Huang et al., 6 Mar 2026).

This motivation is consistent with a broader concern in factuality evaluation: static benchmarks can become unreliable not only because of annotation difficulty but also because of temporal misalignment. A separate study on benchmark aging reported that at least 24% and up to 64% of time-sensitive questions in widely used factuality benchmarks no longer match current real-world facts, and that the temporal alignment gap is positive for 70% of evaluated settings (Jiang et al., 8 Oct 2025). This suggests that DeepFact-Bench’s revisable design addresses one form of benchmark brittleness—evidence-driven correction—within a wider landscape in which static gold labels are increasingly difficult to defend.

2. Audit-then-Score as an evolving benchmarking framework

DeepFact-Bench is built from Evolving Benchmarking via Audit-then-Score. AtS treats benchmarking as a dynamic, co-evolving process rather than a fixed gold-standard labeling exercise. Its core principles are to treat benchmark labels as revisable consensus, to use models as both challengers and auditees, and to rely on human or trusted-agent auditors to adjudicate disagreements using evidence-based rationales (Huang et al., 6 Mar 2026).

At time $t$, the benchmark is formalized as

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$

where $c_i$ is the claim, $d_i$ the DRR context, $y_i^{(t)}$ the current label, and $\rho_i^{(t)}$ the rationale. A challenger model is run on the current benchmark state and produces predictions

$\hat{y}_i = M_t(c_i, d_i).$

Whenever $\hat{y}_i$ disagrees with the current consensus label $y_i^{(t)}$, the challenger must submit a proposed verdict together with an auditable rationale $\hat{\rho}_i$. The auditor compares the challenger’s rationale to the current rationale and accepts the revision if the challenger’s evidence is judged stronger. Accepted revisions are collected as

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$0

after which the benchmark evolves according to

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$1

The paper also presents the state transition more generally as

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$2

Only after the benchmark has been updated is the challenger scored against the refined benchmark state. This sequencing distinguishes AtS from static scoring schemes of the form

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$3

which assume the benchmark labels are already correct. In DeepFact-Bench, scoring is downstream of audit rather than prior to it (Huang et al., 6 Mar 2026).

3. Annotation protocol, auditing, and micro-gold calibration

The seed annotation process recruits verified, active PhD-level domain experts. These experts annotate DRR claims derived from their own research questions, and claim selection is importance- and risk-stratified so that expert effort is concentrated on high-impact, error-prone claims (Huang et al., 6 Mar 2026).

A central design element is the use of hidden micro-gold claims as a continuous external accuracy check. The annotation protocol embeds adversarially constructed, known-answer micro-gold claims comprising 25% of the data. Unsupported micro-golds are created with subtle factual errors at the collection, analysis, or generalization stage, following a detailed taxonomy. Supported micro-golds are claims with narrow scope and explicit citation, validated by LLM and human. This allows the benchmark builders to monitor expert accuracy continuously rather than inferring quality solely from agreement statistics (Huang et al., 6 Mar 2026).

The auditing process proceeds in multiple AtS rounds. Round 0 consists of initial expert-only labels. In Round 1 and later, experts audit increasingly stronger agents, and every agent–human disagreement is resolved by the human auditor. The process continues until micro-gold accuracy stabilizes or the audit budget is exhausted. The reported empirical pattern is that experts are substantially more reliable as auditors than as one-shot labelers, and that most expert effort is spent in the initial round, with later rounds requiring less human attention as the number of remaining disagreements declines (Huang et al., 6 Mar 2026).

A plausible implication is that DeepFact-Bench operationalizes expertise as a corrective process rather than as an infallible initial source of truth. That interpretation follows directly from the measured gap between one-shot expert labeling and expert auditing performance.

4. Benchmark structure, scope, and label semantics

DeepFact-Bench contains 944 claims from 20 DRRs covering six technical domains: Computer Science, Control, Environmental Engineering, Education, Public Health, and Engineering Management. The benchmark is split into 323 claims from 5 reports for validation and 621 claims from 15 reports for test. Within the test set, 143 claims are adversarial micro-golds, and 27% of the remaining test claims are naturally unsupported (Huang et al., 6 Mar 2026).

Component	Count	Notes
Validation	323 claims	5 reports
Test	621 claims	15 reports
Total	944 claims	20 DRRs

Each datapoint is a claim-level tuple paired with document context. The paper describes the task input as $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$4, where $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$5 is the claim sentence, $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$6 is the full DRR context, and $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$7 is the factuality label. The benchmark also associates each claim label with a rationale, yielding a structure of the form $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$8 suitable for detailed document-level verification (Huang et al., 6 Mar 2026).

The label space is

$B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\},$9

The benchmark is also versioned: each audit round yields a new snapshot, such as DeepFact-Bench$c_i$0 and DeepFact-Bench$c_i$1, with detailed changelogs for provenance and comparison. The paper further states that the label definitions are transferable and include propagation rules for composite claims, which supports compatibility and comparison across datasets (Huang et al., 6 Mar 2026).

5. Empirical behavior and benchmark evolution

The most prominent empirical result is the improvement in expert micro-gold accuracy across audit rounds. On the hidden micro-gold set, accuracy rose from 60.8% to 90.9% after four AtS rounds, and report-level paired bootstrap analysis confirmed that the improvements are robust. This result is used to argue that benchmark quality can be materially improved through iterative auditing rather than assumed at the outset (Huang et al., 6 Mar 2026).

DeepFact-Bench is presented together with DeepFact-Eval, a document-level verification agent with a grouped lite variant. On DeepFact-Bench, DeepFact-Eval outperformed existing verifiers: the summary reports accuracy rising from 58% for SAFE, or approximately 69% for deep research agents, to 83.4% and above for DeepFact-Eval. The same summary states that DeepFact-Eval transfers well to external factuality datasets and performs nearly at maximum possible accuracy there, with remaining disagreements judged mostly to reflect benchmark annotation flaws rather than model errors (Huang et al., 6 Mar 2026).

These findings are significant for two related reasons. First, they indicate that benchmark construction quality is not merely a background concern but an active determinant of measured verifier performance. Second, they suggest that co-evolution of benchmarks and agents can improve both evaluation reliability and verifier development. The paper states this directly in terms of reduced annotation noise, high-fidelity ground truth, and support for agent co-evolution as stronger verifiers emerge (Huang et al., 6 Mar 2026).

6. Relation to other factuality benchmarks and benchmark maintenance

DeepFact-Bench occupies a distinct niche within factuality benchmarking. MFC-Bench, for example, is a benchmark for multimodal fact-checking with large vision-LLMs, comprising 33,000+ samples across Manipulation, Out-of-Context, and Veracity Classification. It benchmarks factual accuracy in image–text settings and reports that current models are especially weak on manipulation detection (Wang et al., 2024). DeepFact-Bench, by contrast, targets claim-level factuality in document-scale DRRs, where the central unit is a claim paired with full report context and an auditable rationale.

The benchmark also differs from static factuality testbeds whose main concern is temporal obsolescence. Work on benchmark aging shows that static factuality benchmarks can become outdated rapidly and can therefore mis-evaluate newer models that are more aligned with current facts (Jiang et al., 8 Oct 2025). DeepFact-Bench does not solve temporal drift in general, but its versioned, revisable architecture provides a maintenance mechanism that is absent from fixed-label evaluation.

The maintenance plan is explicit. After public release, the benchmark is intended to continue updating through AtS, using mostly agent auditors with occasional expert validation. This makes DeepFact-Bench a maintained benchmark rather than a one-time release. A common misconception in factuality evaluation is that “gold” labels are immutable once collected; DeepFact-Bench is built around the opposite assumption, namely that labels and rationales should remain contestable when stronger evidence appears (Huang et al., 6 Mar 2026).