DeepFact-Eval: Factual Verification for DRRs

• DeepFact-Eval is a multi-step, literature-grounded verifier that assesses claim-level factuality in deep research reports via iterative evidence retrieval and audit-driven revisions.
• It employs the DeepFact-Bench benchmark and an Audit-then-Score protocol, enabling evolving expert annotations and enhanced claim verification across multiple sources.
• Empirical evaluations show DeepFact-Eval significantly outperforms traditional fact-checkers and deep research baselines, with cost-effective grouped variants offering additional efficiency.

DeepFact-Eval is an agentic, document-level factuality verifier designed for deep research reports (DRRs): long, expert-style, search-augmented reports that synthesize information across many sources. It is introduced together with DeepFact-Bench, a versioned DRR factuality benchmark, and with Evolving Benchmarking via Audit-then-Score (AtS), a protocol in which benchmark labels and rationales are explicitly revisable when a verifier submits better evidence and an auditor accepts the revision (Huang et al., 6 Mar 2026). The central problem is claim-level factuality in DRRs, where existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims, while DRR claims are often multi-hop, distributed across full papers, and sensitive to omitted caveats, overgeneralization, and cross-paper consistency (Huang et al., 6 Mar 2026).

1. Definition, scope, and naming

DeepFact-Eval is defined as a multi-step, literature-grounded verifier for claim-level factuality at the sentence level in DRRs (Huang et al., 6 Mar 2026). Its input is a claim sentence $c_i$ together with the full DRR context $d_i$, and its output is a factuality verdict plus a rationale grounded in retrieved literature. The task uses the labels Supported, Inconclusive, Contradictory, and None for non-verifiable sentences. At reporting time, Contradictory and Inconclusive are merged into Unsupported, yielding a binary supported/unsupported task (Huang et al., 6 Mar 2026).

The system differs from prior fact-checkers such as FactCheck-GPT, SAFE, VeriScore, and FIRE, which the paper characterizes as systems built for atomic, factoid-style claims, snippet-level evidence matching, and single-pass retrieval-and-judge pipelines (Huang et al., 6 Mar 2026). In the DRR setting, the paper argues that citations in the report do not guarantee global correctness or current scientific consensus, and that uncited claims may still be factual. This motivates verification over the entire report and over retrieved literature rather than over local snippets alone (Huang et al., 6 Mar 2026).

A naming ambiguity is present in the broader factuality-evaluation literature represented here. One paper presents GraphEval as “DeepFact-Eval” in summary form, but the method itself is explicitly named GraphEval and evaluates LLM factuality by turning DBpedia triples into declarative statements and using a judge model over hidden states (Liu et al., 2024). By contrast, the explicit system name DeepFact-Eval is introduced in “DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality” and refers to document-level verification for DRRs (Huang et al., 6 Mar 2026).

2. DeepFact-Bench and Audit-then-Score

DeepFact-Bench is the benchmark instantiated by the paper’s new evaluation protocol. It contains 944 claims from 20 reports spanning 6 domains. The split is 323 claims from 5 CS reports for validation and 621 claims from 15 reports across 5 other domains for testing. Among the test claims, 143 claims are micro-golds, 120 of those are adversarially constructed, and excluding adversarial examples, 27.0% of remaining test claims are naturally unsupported. Each benchmark item includes the verbatim claim sentence, its source report/context, a final verdict, and an auditable rationale (Huang et al., 6 Mar 2026).

AtS replaces a static gold set with an evolving benchmark state. The benchmark update rule is:

$B_{t+1} = F(B_t, U_{M,t}, A_t)$

where $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\}$ is the current benchmark state, $U_{M,t} = \{(i,\hat{y}_i,\hat{\rho}_i)\}$ are challenger proposals, and $A_t$ is the auditor. Accepted updates are

$$\Delta B_t = \{(c_i, d_i, \hat{y}_i, \hat{\rho}_i) \mid A_t(\hat{\rho}_i, \rho_i^{(t)}) = ACCEPT\}$$

and benchmark evolution is

$B_{t+1} = B_t \oplus \Delta B_t.$

The corresponding static exact-match score is

$$\mathrm{Score}(M; S) = \frac{1}{N}\sum_i \mathbf{1}\!\left[\hat y_i = y_i^{h}\right].$$

The paper’s claim is that this static regime is brittle in the DRR setting because expert one-shot labels are often unstable (Huang et al., 6 Mar 2026).

The hidden micro-gold study is the empirical basis for that claim. In round 0, unassisted PhD-level specialists achieve only 60.8% accuracy on the hidden micro-gold set. Across four AtS rounds, micro-gold accuracy rises to 90.9%, and the paper interprets this as evidence that experts are substantially more reliable as auditors than as one-shot labelers (Huang et al., 6 Mar 2026). Under an audit-frequency ablation with $p \in \{0.25,0.5,0.75,1.0\}$, round-3 accuracies are 76.2 / 85.3 / 89.5 / 90.9, and a stricter update rule improves round-2 accuracy from 0.86 to 0.88 while slightly lowering round-3 accuracy from 0.909 to 0.902 (Huang et al., 6 Mar 2026). This suggests that benchmark construction is itself an adaptive process in deep research factuality evaluation.

3. Verification pipeline and grouped lite variant

DeepFact-Eval is described as a multi-step agent rather than as a single-pass classifier. It first reads the entire report, unlike narrow-window verifiers such as VeriScore. It then performs breadth-oriented query planning, generating multiple targeted search queries to explore the relevant literature. Retrieved sources are summarized, with GPT-4.1 mini used for summarization to reduce cost. For each document, the system generates depth-oriented follow-up questions to recover claim-critical details that may not appear in summaries. If evidence is insufficient and budget remains, the agent iterates; otherwise it outputs a final verdict and rationale (Huang et al., 6 Mar 2026).

The inference setup uses max steps = 2, max queries = 5, max sources = 40, and max completion tokens = 8192. Verification uses GPT-4.1, while summarization uses GPT-4.1 mini. The paper converts mini token usage into GPT-4.1-equivalent cost using price ratios (Huang et al., 6 Mar 2026).

A grouped lite variant is introduced as DeepFact-Eval lite, also referred to as grouped verification. In this mode, semantically related claims are verified jointly so that shared context and overlapping evidence can be reused. The reported group sizes are Group=5 and Group=10. The design trades some accuracy for substantial cost savings, but remains materially different from snippet-level pipelines because the grouped agent still operates in a literature-review style rather than by direct local entailment (Huang et al., 6 Mar 2026).

The sentence-level label semantics are strict. If any factual part of the sentence is contradicted, the whole sentence is Contradictory. If no contradiction exists but some part is unresolved, the sentence is Inconclusive. If all factual parts are supported, the sentence is Supported. If there are no verifiable factual claims, the sentence is None (Huang et al., 6 Mar 2026). This sentence-level treatment is important because DRR sentences often compress multiple factual commitments into a single synthesis statement.

4. Empirical performance

The main empirical result on DeepFact-Bench is that DeepFact-Eval outperforms both traditional fact-checkers and deep-research-style baselines. The paper reports the following values (Huang et al., 6 Mar 2026):

System	Accuracy / F1	Cost
DeepFact-Eval	83.4 / 86.9	$1.16
DeepFact-Eval (Group=5)	77.9 / 83.1	$0.30
DeepFact-Eval (Group=10)	76.3 / 82.2	$0.21

For comparison, FactCheck-GPT reaches 55.0 accuracy and 58.3 F1, SAFE 55.9 and 53.0, VeriScore 52.5 and 48.9, FIRE 58.5 and 63.2, GPT-Researcher (Deep) 69.1 and 79.7, GPT-Researcher (Deep+) 68.3 and 79.3, and SmolAgents 68.8 and 69.5 (Huang et al., 6 Mar 2026). The paper states that DeepFact-Eval beats the best traditional method by +27.5 accuracy points over SAFE and beats the best deep-research baseline by +14.3 over GPTResearcher. Report-level paired bootstrap yields a 14.7-point advantage over GPT-Researcher with 95% CI: [7.4, 23.3], and a 15.0-point advantage over SmolAgents with 95% CI: [9.5, 20.5] (Huang et al., 6 Mar 2026).

The method is also evaluated with alternate backbones. Reported results are 87.2 acc, 89.9 F1, 87.9 precision, 91.9 recall for GPT-5; 81.5 acc, 85.0 F1, 84.6 precision, 85.3 recall for Gemini-2.5-Pro; and 72.5 acc, 77.4 F1, 78.1 precision, 76.6 recall for Qwen-3-32B (Huang et al., 6 Mar 2026). These numbers indicate that the agentic verification procedure is not tied to a single backbone, though the paper does not claim backbone invariance.

External evaluation is reported on SciFact, ExpertQA, and Factcheck-Bench. DeepFact-Eval disagrees with the benchmark on 29/188 SciFact instances, 90/200 ExpertQA cases, and 36/200 Factcheck-Bench claims. After audit and re-annotation, the paper estimates 94.7% accuracy on SciFact and 93.0% on Factcheck-Bench, while noting that in a blinded re-annotation subset for ExpertQA, experts side with DeepFact-Eval in 28/30 cases (Huang et al., 6 Mar 2026). The stated interpretation is that many residual disagreements arise from annotation noise, ambiguity, evidence–label misalignment, or non-verifiable discourse rather than from straightforward model error.

5. Position within factuality-evaluation research

DeepFact-Eval occupies a distinct position relative to other factuality-evaluation paradigms in the provided literature. GraphEval evaluates LLM factuality at scale by converting DBpedia triples into declarative statements and using a lightweight judge model over hidden states to predict True, False, or I don’t know (IDK). Its testbed is DBpedia with 4,928,232 entities, 633 relations, and 16,915,848 triples, and its emphasis is scale, efficiency, and a multi-metric view of Correctness, Truthfulness, and Informativeness (Liu et al., 2024). DeepFact-Eval, by contrast, verifies sentence-level claims in DRRs through iterative literature retrieval and auditing rather than through white-box knowledge-state estimation over a large knowledge graph (Huang et al., 6 Mar 2026).

FaaF addresses factual recall evaluation in retrieval-augmented generation by replacing prompt-based fact verification with function-calling over typed outputs. It is motivated by the observation that prompt-based verification is unreliable under incomplete or inaccurate references, and it reduces LM calls by verifying multiple facts in a single function call (Katranidis et al., 2024). FaStFact targets long-form LLM generations by improving claim extraction, evidence collection, and scoring, with chunk-level extraction, confidence-based pre-verification, and document-level evidence from crawled webpages (Wan et al., 13 Oct 2025). Both are relevant antecedents, but neither is framed around DRRs, evolving benchmarks, or document-level claim verification in the literature-review setting.

The deep-research benchmark most closely aligned in spirit is DR$d_i$0-Eval, which evaluates deep research agents on multimodal, multi-file report generation under a per-task static sandbox corpus. Its metrics include Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and the paper reports Pearson $d_i$1, Spearman $d_i$2, and pairwise agreement = 0.89 against human judgments (Xie et al., 16 Apr 2026). A plausible implication is that DeepFact-Eval and DR$d_i$3-Eval address adjacent levels of the same problem: the former centers on claim-level verification for DRRs, whereas the latter evaluates full report generation under reproducible multimodal retrieval conditions.

The broader evaluation ecosystem adds two further reference points. TaskEval / GenValidator synthesizes task-specific evaluation programs when no benchmark, metric, or labeled dataset exists, using a task-agnostic meta-model, an interaction protocol for human feedback, and an eval synthesiser (Widanapathiranage et al., 4 Dec 2025). Eval Factsheets proposes structured documentation for evaluations across Context, Scope, Structure, Method, and Alignment, with the explicit goal of improving reproducibility, transparency, and informed decision-making (Bordes et al., 3 Dec 2025). These frameworks are not factuality metrics, but they are directly relevant to documenting or operationalizing DeepFact-Eval-style systems.

6. Limitations, failure modes, and implications

The paper is explicit that DeepFact-Eval remains limited to literature verification. It does not perform direct experimental validation, and it cannot verify claims that require new data, empirical lab work, or settings in which the literature is silent or conflicting (Huang et al., 6 Mar 2026). Computational cost is also a stated limitation: even with a lite variant, deep verification remains expensive because it requires long-context reasoning and iterative retrieval (Huang et al., 6 Mar 2026).

Failure cases include missing critical evidence due to incomplete retrieval, retrieving close evidence but missing nuances, and failing to verify niche subclaims in long sentences (Huang et al., 6 Mar 2026). These are characteristic failure modes for document-level verification, especially when claims summarize across many sources or compress several factual commitments into one sentence.

The paper’s broader implication is methodological rather than only algorithmic: as deep research agents improve, static gold labels become the bottleneck. In this view, what appears to be model error may instead reflect annotation divergence, ambiguous claims, insufficient evidence in the benchmark, or mislabeled and mis-scoped examples (Huang et al., 6 Mar 2026). This interpretation is reinforced by the micro-gold study and by the post-audit improvements in external benchmark accuracy estimates.

In that sense, DeepFact-Eval is not only a verifier but also part of a proposal about how factuality benchmarks should be maintained in expert, literature-heavy domains. The combination of a document-level verification agent, a versioned benchmark with auditable rationales, and an Audit-then-Score protocol defines a fact-centric evaluation regime in which both the model and the benchmark can be revised in light of stronger evidence (Huang et al., 6 Mar 2026).