DeepSearchQA: Benchmark Comprehensiveness Gap Evaluation Framework

Updated 8 September 2026

Its tasks require mapping an information landscape to produce complete, precise, and verifiable answer sets, distinguishing it from conventional question-answering benchmarks that typically reward finding a single correct answer.
DeepSearchQA spans 17 fields, with representative areas including politics and government, finance and economics, science, health, etc., and utilizes 900 hand-crafted prompts grounded in the open web.
The benchmark evaluates on completeness, precision, and justified stopping criteria, emphasizing both retrieval and reasoning to ensure high quality in complex answer sets.

DeepSearchQA is a benchmark for evaluating autonomous web-research agents on difficult multi-step information-seeking tasks that require exhaustive answer-set construction rather than retrieval of a single fact. It contains 900 hand-crafted prompts across 17 fields, grounded in the open web and paired with objectively verifiable answer sets. Its central evaluation problem is the “Comprehensiveness Gap”: an agent may produce factually correct claims while omitting valid entities, including duplicates or invalid candidates, failing to resolve dependencies, or stopping without adequate evidence (Gupta et al., 28 Jan 2026). The name should be distinguished from Deep Search DocQA, an IBM Research application for conversational extraction from ESG disclosure reports, which is a system description rather than a benchmark named DeepSearchQA (Mishra et al., 2023).

1. Definition and scope

DeepSearchQA evaluates whether an agent can map an information landscape and return a complete, precise, and verifiable answer set. Unlike conventional question-answering benchmarks that typically reward finding one correct answer, its tasks may require systematic collation from disparate sources, multi-step filtering, entity resolution, constraint satisfaction, and a justified stopping decision.

The benchmark spans 17 fields, with representative areas including politics and government, finance and economics, science, health, history, geography, and media. Prompts are grounded in the open web and use curated ground truth. Temporal anchoring is common: tasks may refer to a specified census year, fiscal-year end, CDC or Eurostat data, NHTSA statistics, Statistics Canada, or a company report. This design reduces temporal drift in the intended answer, although web pages can still change or disappear.

DeepSearchQA distinguishes between two broad answer types:

Single-answer tasks: one entity or value is correct, such as a name or date.
Set-answer tasks: several entities or values must be returned, including enumeration and composite responses.

For set-answer tasks, an answer is successful only when it contains every valid item and excludes invalid, duplicate, adjacent, or merely plausible items. Accordingly, the benchmark evaluates not only retrieval and reasoning but also termination: an agent must distinguish “no additional item has yet been found” from “no additional item satisfies the conditions.”

The benchmark is available through Kaggle with an open leaderboard at https://www.kaggle.com/benchmarks/google/dsqa/leaderboard (Gupta et al., 28 Jan 2026). DeepSearchQA is therefore an evaluation benchmark, not the ESG report collection described in Deep Search DocQA, not the HDS-QA synthetic dataset, and not a generic name for all deep-search question-answering systems.

2. Task construction and dependency structure

The 900 prompts were curated by expert annotators from varied information sources. Prompt design sought tasks that were realistic, objective, open-web grounded, sufficiently complex for multi-step retrieval, and stable enough for reproducible evaluation.

Tasks are structured as causal or dependency chains. Later filtering or reasoning operations depend on intermediate results from earlier stages. Representative structures include:

$\text{Population filter} \rightarrow \text{Life-expectancy filter} \rightarrow \text{Immunization filter} \rightarrow \text{Final intersection}$

and:

$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$

This organization requires an agent to preserve intermediate entities, dates, units, rankings, exclusions, and source-specific definitions across a long trajectory. It also creates error propagation: failure to identify an entity or apply an early constraint can prevent later valid answers from being discovered.

Representative task roles include:

Epidemiologist: identify U.S. states satisfying specified death-rate conditions in 2019 and 2020.
Video Gamer: identify hardware families exceeding 100 million sales, exclude handhelds, filter by CPU bit width, and inspect parent-company assets using official financial reports.
Safety Auditor: identify states with NHTSA fatality rates below one per 100 million vehicle miles traveled for at least two consecutive years between 2000 and 2005.
Demographer: combine Statistics Canada measures involving population centers, GDP growth, and Aboriginal identity populations.

The taxonomy groups tasks into three diagnostic cognitive categories:

Structured Retrieval (“The Search”): multi-step search, obscure information, and dependency-graph retrieval.
Context Management (“The Assembly”): combining heterogeneous pages or documents while retaining relevant state.
Logical Reasoning (“The Thinker”): constraint solving, synthesis, deduction, and conflict resolution.

These categories are diagnostic rather than separate scoring tracks in the reported results.

Ground truth is established through a three-stage protocol. Three reviewers independently solve each prompt without access to the curated answer; their results are compared with the curator’s ground truth; and discrepancies are investigated. Ambiguous or flawed prompts are filtered out. This procedure is intended to establish an objectively verifiable answer set rather than merely a plausible reference response.

3. Evaluation protocol and metrics

For prompt $i$ , let $S_i$ be the set of distinct answers submitted by an agent and $G_i$ the ground-truth answer set. Evaluation is based on semantic membership rather than answer order.

Precision is:

$P_i = \frac{|S_i \cap G_i|}{|S_i|}.$

Recall is:

$R_i = \frac{|S_i \cap G_i|}{|G_i|}.$

The primary ranking metric is F1:

$F_{1,i} = 2\cdot \frac{P_iR_i}{P_i+R_i}.$

F1 penalizes both under-retrieval and over-retrieval. For a single-answer task, it reduces to an exact-match-style binary success measure. For set-answer tasks, it rewards answers that are simultaneously exhaustive and precise.

Responses are also assigned to categorical set relationships:

Fully correct: $S_i=G_i$ .
Fully incorrect: $S_i\cap G_i=\emptyset$ .
Partially correct: some correct answers are found, but the complete ground-truth set is not recovered; extra incorrect answers may also be present.
Correct with extraneous answers: $\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 0, meaning all correct items are returned but at least one invalid item is added.

For single-answer questions, submitting multiple candidates is not accepted as a successful answer. “Brazil and Italy” does not correctly answer “Who won the 1994 World Cup?” even though Brazil is included.

The benchmark uses Gemini 2.5 Flash as an LLM-based semantic judge. For each submitted item $\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 1, the evaluator checks semantic equivalence against ground-truth items $\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 2, records whether expected answer components were found, identifies excessive answers, and produces an explanation. The main results report percentages with 95% confidence intervals, although the supplied description does not specify the interval construction or formal statistical tests.

The benchmark’s scoring design exposes a distinction between high partial quality and exact-set success. An agent can achieve high F1 by retrieving most valid items while still failing to recover obscure tail entities or by adding unsupported candidates. This distinction is central to DeepSearchQA’s evaluation philosophy.

4. Reported benchmark results

The principal comparison includes deep-research agents, general reasoning systems, and standalone models. The reported table is:

Rank	Model	Fully correct	Fully incorrect	Correct with extraneous answers	F1
1	Gemini Deep Research Agent	66.09	9.95	10.30	81.90
2	GPT-5 Pro High Reasoning	65.18	14.13	8.12	78.98
3	GPT-5 High Reasoning	59.41	19.91	6.56	73.24
4	Gemini 3 Pro Preview	56.56	12.78	9.89	76.86
5	o3 Deep Research	44.24	20.09	11.74	66.45
6	o4 Mini Deep Research	40.36	24.19	7.80	61.76
7	Gemini 2.5 Flash	25.92	45.27	5.90	42.99
8	Claude 4.5 Opus	24.01	50.66	4.18	40.20
9	Claude 4.5 Sonnet	16.04	64.25	2.90	27.85
10	Claude 4.5 Haiku	12.78	71.00	1.89	22.24

Gemini Deep Research Agent has the highest reported F1, at 81.90, followed by GPT-5 Pro High Reasoning at 78.98. The paper states that their fully correct rates are statistically comparable, although formal test details are not supplied. Deep-research systems substantially outperform smaller or less agentic models, while Gemini 2.5 Flash and Claude 4.5 Haiku have fully incorrect rates of 45.27 and 71.00, respectively.

The results demonstrate that strong partial retrieval does not imply exact-set completion. Gemini Deep Research obtains F1 of 81.90 but is fully correct on 66.09% of prompts; GPT-5 Pro obtains F1 of 78.98 and is fully correct on 65.18%. The remaining gap reflects missed long-tail entities, extraneous candidates, incomplete dependency handling, entity-resolution errors, and premature stopping.

Additional test-time sampling improves fully correct performance in the reported experiment:

$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 3: 67.18%.
$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 4: 74.51%.
$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 5: 81.72%.
$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 6: 85.71%.

The supplied description does not specify the aggregation method or corresponding changes in precision and F1.

5. Failure modes and agent requirements

DeepSearchQA identifies several recurring failure modes that differ from ordinary hallucination.

Premature stopping and under-retrieval occur when an agent finds several obvious answers and stops before investigating less prominent valid entities. This reduces recall and is especially damaging when later dependency steps can reveal additional candidates.

Hedging and over-retrieval occur when an agent lists multiple candidates rather than resolving the question. In a single-answer task, “Canberra or Sydney” is incorrect. In a set-answer task, the analogous failure is returning the correct set plus plausible but invalid items.

Low-confidence over-retrieval occurs when an agent expands the candidate list with weakly supported entities to raise recall. This degrades precision and can lower F1 even when coverage increases.

Quantitative estimation and synthesis errors arise when retrieval succeeds but the agent fails to calculate, rank, or compare the relevant values. The paper describes a case in which an agent identifies all qualifying countries but estimates transport tonnage rather than obtaining concrete values required for ranking.

Tool and document-recovery failures occur when relevant data are found in inaccessible formats, such as an Excel file that the agent cannot open. Successful research therefore requires recovery through alternate sources or extraction methods.

Search-planning failures arise when an agent follows an incorrect branch and fails to recover. Smaller systems may diverge into irrelevant search trajectories and return no valid answers.

Missed dependencies and constraint violations occur when the agent retrieves the relevant entities but fails to apply a final exclusion, temporal condition, ranking requirement, or set difference.

Entity-resolution errors occur when aliases, abbreviations, variants, or related members are treated as separate entities. This inflates answer sets and reduces precision.

These failure modes imply that a high-performing DeepSearchQA agent requires systematic exploration, candidate ledgers, entity normalization, provenance tracking, constraint preservation, contradiction handling, robust tool recovery, and dynamic stopping. Raw language-model knowledge or internal reasoning without search control is insufficient.

Several later systems use DeepSearchQA or closely related deep-search tasks to study distinct capabilities.

Search planning and hybrid parallelism. HybridDeepSearcher is trained on HDS-QA, a synthetic dataset of hybrid-hop questions combining parallelizable independent subqueries with sequentially dependent subqueries. It reports F1 of 44.1 on FanOutQA and 17.2 on BrowseComp-50, but HDS-QA is not DeepSearchQA and the paper does not redefine the benchmark (Ko et al., 26 Aug 2025).

Multi-turn reinforcement learning. DeepDive constructs difficult questions from KILT and AMiner knowledge graphs and trains browsing agents with multi-turn RL. DeepDive-32B RL reports 14.8 on BrowseComp, but the supplied results do not provide a DeepSearchQA score (Lu et al., 12 Sep 2025).

Epistemic verification. The Epistemic Ledger and LiveLedger track objective evidence and agent belief separately for each constraint. In an evaluation containing 43 DeepSearchQA questions within a 215-question multi-constraint set, LiveLedger improves ReAct-120B accuracy from 39.1 to 50.7 and reduces underverified answer rate from 76.3 to 49.8 (Ko et al., 7 Feb 2026).

Stochasticity and reproducibility. A study using 20 DeepSearchQA instances and 10 repeated executions per instance reports that structured intermediate outputs and consensus-based query generation reduce run-to-run variation. The combined method raises extracted-answer accuracy from 0.24 to 0.36 while reducing average variance from 0.69 to 0.47; the paper’s headline reduction of 22% differs from the approximately 32% reduction implied by its displayed table (Zhai et al., 26 Feb 2026).

Context management. SelfCompact evaluates adaptive context compaction on 150-question subsets of DeepSearchQA. For GLM-4.7-Flash, MiniMax-M2.5, and MiMo-V2-Flash, SelfCompact obtains accuracies of 44.0, 61.3, and 56.8, respectively, compared with no-compaction scores of 34.2, 52.0, and 46.5 (Li et al., 22 Jun 2026).

Small-agent training. DR-Venus-4B scores 37.7 with SFT and 39.6 with RL on DeepSearchQA. Its training uses cleaned long-horizon trajectories, resampling, and IGPO turn-level information-gain rewards (Team et al., 21 Apr 2026).

Recursive refinement. AREX reports DeepSearchQA F1 of 78.5 for a dense 4B model and 89.9 for a 122B-total, 10B-activated MoE model. Its architecture alternates inner research with outer constraint-wise auditing and targeted follow-up research (Lu et al., 23 Jul 2026).

Harness evolution. JIT-Agent generates task-adaptive harnesses for frozen backbone models. On DeepSearchQA, it reports 85.1 for DeepSeek-V4-Flash with JIT-Agent, compared with 76.2 for the vanilla scaffold, and 93.9 for GLM-5.2 with JIT-Agent, compared with 89.2 for GLM-5.2 alone. A controlled DeepSeek-V4-Flash comparison reports 85.1 for JIT-Agent versus 80.4 for NanoBot, 79.6 for Claude Code, 77.8 for Codex, and 75.9 for OpenCode (Zhang et al., 26 Aug 2026).

These systems address different layers of the benchmark problem: retrieval scheduling, long-horizon persistence, epistemic verification, stochasticity, context compression, small-model training, recursive answer refinement, and harness design. Their results should not be conflated because they use different models, tools, evaluation subsets, metrics, and experimental conditions.

7. Limitations and research significance

DeepSearchQA’s outcome-focused design provides a flexible comparison across architectures, but it also limits process diagnosis. Aggregate answer scores do not reveal whether an error arose from query generation, source selection, retrieval incompleteness, entity resolution, evidence synthesis, contradiction handling, or stopping.

The benchmark does not, in the supplied description, provide detailed trajectory logs, token budgets, browsing traces, source-level annotations, or a controlled classical information-retrieval baseline. LLM-as-a-judge evaluation introduces possible bias, including sensitivity to wording, answer length, semantic equivalence, and judge-model capabilities. The open-web setting introduces temporal drift, page disappearance, inaccessible documents, ranking changes, and source instability.

A further limitation is that exact-set evaluation does not by itself assess citation entailment or evidence provenance. An agent may produce the correct set through a weakly grounded path, while another may produce a partially correct set with stronger evidence. DeepSearchQA’s primary F1 and categorical outcomes do not fully distinguish these cases.

Complementary benchmarks address some of these gaps. SeekerGym evaluates document-level information-gathering completeness and uncertainty calibration, reporting best ordinary-belief completeness of 42.5% on Wikipedia and 29.2% on machine-learning surveys (Kim et al., 18 Apr 2026). DRACO evaluates open-ended reports along factual accuracy, breadth and depth, presentation quality, and citation quality, thereby complementing DeepSearchQA’s more closed-ended answer-set evaluation (Zhong et al., 12 Feb 2026). The Epistemic Ledger work further suggests measuring whether every explicit constraint is supported before an agent terminates (Ko et al., 7 Feb 2026).

DeepSearchQA’s principal significance is evaluative: it changes the objective from finding a correct answer to producing a complete, clean, verifiable set under open-world uncertainty. Its central performance tension is:

$\text{Retrieve macroeconomic data} \rightarrow \text{Retrieve GDP data} \rightarrow \text{Calculate annual growth} \rightarrow \text{Intersect threshold conditions}.$ 7

Future DeepSearchQA-oriented evaluations are likely to benefit from process-level diagnostics alongside final F1, including evidence coverage, citation entailment, contradiction-recognition rate, candidate rejection accuracy, entity-resolution quality, premature-exit rate, stagnation, search cost, latency, and calibrated estimates of remaining undiscovered information. The benchmark’s central challenge remains the same: a deep-research agent must not merely find plausible answers, but systematically explore, verify, deduplicate, synthesize, and know when its research is complete.