---
title: 'DeepFact-Eval: Factual Verification for DRRs'
url: https://www.emergentmind.com/topics/deepfact-eval
type: topic
---

# DeepFact-Eval: Factual Verification for DRRs

DeepFact-Eval is an agentic, document-level factuality verifier designed for deep research reports (DRRs): long, expert-style, search-augmented reports that synthesize information across many sources. It is introduced together with DeepFact-Bench, a versioned DRR factuality benchmark, and with Evolving Benchmarking via Audit-then-Score (AtS), a protocol in which benchmark labels and rationales are explicitly revisable when a verifier submits better evidence and an auditor accepts the revision [2603.05912]. The central problem is claim-level factuality in DRRs, where existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims, while DRR claims are often multi-hop, distributed across full papers, and sensitive to omitted caveats, overgeneralization, and cross-paper consistency [2603.05912].

## 1. Definition, scope, and naming

DeepFact-Eval is defined as a multi-step, literature-grounded verifier for claim-level factuality at the sentence level in DRRs [2603.05912]. Its input is a claim sentence $c_i$ together with the full DRR context $d_i$, and its output is a factuality verdict plus a rationale grounded in retrieved literature. The task uses the labels **Supported**, **Inconclusive**, **Contradictory**, and **None** for non-verifiable sentences. At reporting time, **Contradictory** and **Inconclusive** are merged into **Unsupported**, yielding a binary supported/unsupported task [2603.05912].

The system differs from prior fact-checkers such as FactCheck-GPT, SAFE, VeriScore, and FIRE, which the paper characterizes as systems built for atomic, factoid-style claims, snippet-level evidence matching, and single-pass retrieval-and-judge pipelines [2603.05912]. In the DRR setting, the paper argues that citations in the report do not guarantee global correctness or current scientific consensus, and that uncited claims may still be factual. This motivates verification over the entire report and over retrieved literature rather than over local snippets alone [2603.05912].

A naming ambiguity is present in the broader factuality-evaluation literature represented here. One paper presents **GraphEval** as “DeepFact-Eval” in summary form, but the method itself is explicitly named GraphEval and evaluates LLM factuality by turning DBpedia triples into declarative statements and using a judge model over hidden states [2404.00942]. By contrast, the explicit system name **DeepFact-Eval** is introduced in “DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality” and refers to document-level verification for DRRs [2603.05912].

## 2. DeepFact-Bench and Audit-then-Score

DeepFact-Bench is the benchmark instantiated by the paper’s new evaluation protocol. It contains **944 claims** from **20 reports** spanning **6 domains**. The split is **323 claims from 5 CS reports** for validation and **621 claims from 15 reports across 5 other domains** for testing. Among the test claims, **143 claims are micro-golds**, **120 of those are adversarially constructed**, and excluding adversarial examples, **27.0%** of remaining test claims are naturally unsupported. Each benchmark item includes the verbatim claim sentence, its source report/context, a final verdict, and an auditable rationale [2603.05912].

AtS replaces a static gold set with an evolving benchmark state. The benchmark update rule is:
$$
B_{t+1} = F(B_t, U_{M,t}, A_t)
$$
where $B_t = \{(c_i, d_i, y_i^{(t)}, \rho_i^{(t)})\}$ is the current benchmark state, $U_{M,t} = \{(i,\hat{y}_i,\hat{\rho}_i)\}$ are challenger proposals, and $A_t$ is the auditor. Accepted updates are
$$
\Delta B_t = \{(c_i, d_i, \hat{y}_i, \hat{\rho}_i) \mid A_t(\hat{\rho}_i, \rho_i^{(t)}) = ACCEPT\}
$$
and benchmark evolution is
$$
B_{t+1} = B_t \oplus \Delta B_t.
$$
The corresponding static exact-match score is
$$
\mathrm{Score}(M; S) = \frac{1}{N}\sum_i \mathbf{1}\!\left[\hat y_i = y_i^{h}\right].
$$
The paper’s claim is that this static regime is brittle in the DRR setting because expert one-shot labels are often unstable [2603.05912].

The hidden micro-gold study is the empirical basis for that claim. In round 0, unassisted PhD-level specialists achieve only **60.8%** accuracy on the hidden micro-gold set. Across four AtS rounds, micro-gold accuracy rises to **90.9%**, and the paper interprets this as evidence that experts are substantially more reliable as auditors than as one-shot labelers [2603.05912]. Under an audit-frequency ablation with $p \in \{0.25,0.5,0.75,1.0\}$, round-3 accuracies are **76.2 / 85.3 / 89.5 / 90.9**, and a stricter update rule improves round-2 accuracy from **0.86 to 0.88** while slightly lowering round-3 accuracy from **0.909 to 0.902** [2603.05912]. This suggests that benchmark construction is itself an adaptive process in deep research factuality evaluation.

## 3. Verification pipeline and grouped lite variant

DeepFact-Eval is described as a multi-step agent rather than as a single-pass classifier. It first reads the entire report, unlike narrow-window verifiers such as VeriScore. It then performs breadth-oriented query planning, generating multiple targeted search queries to explore the relevant literature. Retrieved sources are summarized, with **GPT-4.1 mini** used for summarization to reduce cost. For each document, the system generates depth-oriented follow-up questions to recover claim-critical details that may not appear in summaries. If evidence is insufficient and budget remains, the agent iterates; otherwise it outputs a final verdict and rationale [2603.05912].

The inference setup uses **max steps = 2**, **max queries = 5**, **max sources = 40**, and **max completion tokens = 8192**. Verification uses **GPT-4.1**, while summarization uses **GPT-4.1 mini**. The paper converts mini token usage into GPT-4.1-equivalent cost using price ratios [2603.05912].

A grouped lite variant is introduced as **DeepFact-Eval lite**, also referred to as grouped verification. In this mode, semantically related claims are verified jointly so that shared context and overlapping evidence can be reused. The reported group sizes are **Group=5** and **Group=10**. The design trades some accuracy for substantial cost savings, but remains materially different from snippet-level pipelines because the grouped agent still operates in a literature-review style rather than by direct local entailment [2603.05912].

The sentence-level label semantics are strict. If any factual part of the sentence is contradicted, the whole sentence is **Contradictory**. If no contradiction exists but some part is unresolved, the sentence is **Inconclusive**. If all factual parts are supported, the sentence is **Supported**. If there are no verifiable factual claims, the sentence is **None** [2603.05912]. This sentence-level treatment is important because DRR sentences often compress multiple factual commitments into a single synthesis statement.

## 4. Empirical performance

The main empirical result on DeepFact-Bench is that DeepFact-Eval outperforms both traditional fact-checkers and deep-research-style baselines. The paper reports the following values [2603.05912]:

| System | Accuracy / F1 | Cost |
|---|---|---|
| DeepFact-Eval | **83.4 / 86.9** | **$1.16** |
| DeepFact-Eval (Group=5) | **77.9 / 83.1** | **$0.30** |
| DeepFact-Eval (Group=10) | **76.3 / 82.2** | **$0.21** |

For comparison, FactCheck-GPT reaches **55.0** accuracy and **58.3** F1, SAFE **55.9** and **53.0**, VeriScore **52.5** and **48.9**, FIRE **58.5** and **63.2**, GPT-Researcher (Deep) **69.1** and **79.7**, GPT-Researcher (Deep+) **68.3** and **79.3**, and SmolAgents **68.8** and **69.5** [2603.05912]. The paper states that DeepFact-Eval beats the best traditional method by **+27.5 accuracy points over SAFE** and beats the best deep-research baseline by **+14.3 over GPTResearcher**. Report-level paired bootstrap yields a **14.7-point** advantage over GPT-Researcher with **95% CI: [7.4, 23.3]**, and a **15.0-point** advantage over SmolAgents with **95% CI: [9.5, 20.5]** [2603.05912].

The method is also evaluated with alternate backbones. Reported results are **87.2 acc, 89.9 F1, 87.9 precision, 91.9 recall** for **GPT-5**; **81.5 acc, 85.0 F1, 84.6 precision, 85.3 recall** for **Gemini-2.5-Pro**; and **72.5 acc, 77.4 F1, 78.1 precision, 76.6 recall** for **Qwen-3-32B** [2603.05912]. These numbers indicate that the agentic verification procedure is not tied to a single backbone, though the paper does not claim backbone invariance.

External evaluation is reported on **SciFact**, **ExpertQA**, and **Factcheck-Bench**. DeepFact-Eval disagrees with the benchmark on **29/188** SciFact instances, **90/200** ExpertQA cases, and **36/200** Factcheck-Bench claims. After audit and re-annotation, the paper estimates **94.7%** accuracy on SciFact and **93.0%** on Factcheck-Bench, while noting that in a blinded re-annotation subset for ExpertQA, experts side with DeepFact-Eval in **28/30** cases [2603.05912]. The stated interpretation is that many residual disagreements arise from annotation noise, ambiguity, evidence–label misalignment, or non-verifiable discourse rather than from straightforward model error.

## 5. Position within factuality-evaluation research

DeepFact-Eval occupies a distinct position relative to other factuality-evaluation paradigms in the provided literature. **GraphEval** evaluates LLM factuality at scale by converting DBpedia triples into declarative statements and using a lightweight judge model over hidden states to predict **True**, **False**, or **I don’t know (IDK)**. Its testbed is **DBpedia** with **4,928,232 entities**, **633 relations**, and **16,915,848 triples**, and its emphasis is scale, efficiency, and a multi-metric view of **Correctness**, **Truthfulness**, and **Informativeness** [2404.00942]. DeepFact-Eval, by contrast, verifies sentence-level claims in DRRs through iterative literature retrieval and auditing rather than through white-box knowledge-state estimation over a large knowledge graph [2603.05912].

**FaaF** addresses factual recall evaluation in retrieval-augmented generation by replacing prompt-based fact verification with function-calling over typed outputs. It is motivated by the observation that prompt-based verification is unreliable under incomplete or inaccurate references, and it reduces LM calls by verifying multiple facts in a single function call [2403.03888]. **FaStFact** targets long-form LLM generations by improving claim extraction, evidence collection, and scoring, with chunk-level extraction, confidence-based pre-verification, and document-level evidence from crawled webpages [2510.12839]. Both are relevant antecedents, but neither is framed around DRRs, evolving benchmarks, or document-level claim verification in the literature-review setting.

The deep-research benchmark most closely aligned in spirit is **DR$^{3}$-Eval**, which evaluates deep research agents on multimodal, multi-file report generation under a per-task static sandbox corpus. Its metrics include **Information Recall**, **Factual Accuracy**, **Citation Coverage**, **Instruction Following**, and **Depth Quality**, and the paper reports **Pearson $r = 0.78$**, **Spearman $\rho = 0.73$**, and **pairwise agreement = 0.89** against human judgments [2604.14683]. A plausible implication is that DeepFact-Eval and DR$^{3}$-Eval address adjacent levels of the same problem: the former centers on claim-level verification for DRRs, whereas the latter evaluates full report generation under reproducible multimodal retrieval conditions.

The broader evaluation ecosystem adds two further reference points. **TaskEval / GenValidator** synthesizes task-specific evaluation programs when no benchmark, metric, or labeled dataset exists, using a task-agnostic meta-model, an interaction protocol for human feedback, and an eval synthesiser [2512.04442]. **Eval Factsheets** proposes structured documentation for evaluations across **Context**, **Scope**, **Structure**, **Method**, and **Alignment**, with the explicit goal of improving reproducibility, transparency, and informed decision-making [2512.04062]. These frameworks are not factuality metrics, but they are directly relevant to documenting or operationalizing DeepFact-Eval-style systems.

## 6. Limitations, failure modes, and implications

The paper is explicit that DeepFact-Eval remains limited to literature verification. It does not perform direct experimental validation, and it cannot verify claims that require new data, empirical lab work, or settings in which the literature is silent or conflicting [2603.05912]. Computational cost is also a stated limitation: even with a lite variant, deep verification remains expensive because it requires long-context reasoning and iterative retrieval [2603.05912].

Failure cases include missing critical evidence due to incomplete retrieval, retrieving close evidence but missing nuances, and failing to verify niche subclaims in long sentences [2603.05912]. These are characteristic failure modes for document-level verification, especially when claims summarize across many sources or compress several factual commitments into one sentence.

The paper’s broader implication is methodological rather than only algorithmic: as deep research agents improve, static gold labels become the bottleneck. In this view, what appears to be model error may instead reflect annotation divergence, ambiguous claims, insufficient evidence in the benchmark, or mislabeled and mis-scoped examples [2603.05912]. This interpretation is reinforced by the micro-gold study and by the post-audit improvements in external benchmark accuracy estimates.

In that sense, DeepFact-Eval is not only a verifier but also part of a proposal about how factuality benchmarks should be maintained in expert, literature-heavy domains. The combination of a document-level verification agent, a versioned benchmark with auditable rationales, and an Audit-then-Score protocol defines a fact-centric evaluation regime in which both the model and the benchmark can be revised in light of stronger evidence [2603.05912].

Source: https://www.emergentmind.com/topics/deepfact-eval