AutoResearchBench: Scientific Literature Discovery

• AutoResearchBench is a benchmark for autonomous scientific literature discovery that evaluates AI agents’ ability to identify research papers under complex, full-text constraints.
• Deep Research tasks test agents’ ability to extract subtle methodological clues from full-text content to uniquely identify target papers and appropriately abstain when needed.
• Wide Research tasks require retrieving complete sets of papers meeting multi-constraint queries, evaluated through intersection over union to capture both precision and recall.

Searching arXiv for the benchmark paper and closely related deep-research benchmarks to ground the article in current literature. AutoResearchBench is a benchmark for autonomous scientific literature discovery: the evaluation of whether AI agents can search a large scientific corpus, reason over full papers, and either identify a uniquely specified target paper or recover the complete set of papers satisfying a technical specification. Built over a controlled corpus of more than 3 million arXiv papers through the DeepXiv platform, it contains 1,000 total problems—600 Deep Research and 400 Wide Research—and is designed to be research-oriented, literature-focused, and open-ended. Its central empirical result is that the benchmark is extremely difficult: the best reported scores are 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, despite strong performance by frontier models on general agentic browsing benchmarks (Xiong et al., 28 Apr 2026).

1. Research problem and benchmark rationale

AutoResearchBench targets a capability that is central to autonomous research but not well captured by generic web-search or browsing evaluations: finding the right scientific literature under complex, multi-constraint, paper-level conditions. The benchmark formalizes literature discovery as a setting in which the relevant evidence is often not recoverable from titles, abstracts, or snippets alone, but instead resides in method sections, ablation tables, figure captions, appendices, citation chains, author affiliations, and experimental details (Xiong et al., 28 Apr 2026).

This focus distinguishes the benchmark from tasks that reduce research to short-answer retrieval or web-visible evidence. The benchmark’s stated motivation is that current benchmarks such as GAIA, BrowseComp, and related agentic web-browsing tasks do not adequately test paper-level, research-oriented, open-ended literature discovery. In this setting, an agent must reason about correctness, completeness, abstention, and stopping criteria, rather than merely ranking plausible documents (Xiong et al., 28 Apr 2026).

The paper characterizes AutoResearchBench along three dimensions. It is research-oriented because the tasks require scientific understanding and technical reasoning; literature-focused because decisive evidence is usually embedded in full-text paper content; and open-ended because the answer set size is unknown and may be one, zero, or many. A common misconception is that the benchmark is simply a harder form of scholarly search. The design suggests a narrower and more demanding interpretation: it is a test of whether an agent can maintain hypothesis state over long search trajectories while discriminating between near-miss papers that differ only in subtle full-text constraints (Xiong et al., 28 Apr 2026).

2. Task families: Deep Research and Wide Research

AutoResearchBench is organized around two complementary task types, each with a distinct answer-space structure and evaluation target (Xiong et al., 28 Apr 2026).

Task type	Gold answer structure	Primary metric
Deep Research	$|Y^*(q)| \in \{0,1\}$	Exact-match accuracy
Wide Research	$Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$	Intersection over Union

Deep Research

Deep Research evaluates whether an agent can precisely identify one target paper from obfuscated, multi-step, full-text clues. The benchmark defines the gold set as satisfying

$|Y^*(q)| \in \{0,1\}.$

A query therefore has either one unique correct paper or no correct paper. Given a query $q$ and corpus $\mathcal{D}$, the agent predicts a set $\hat{Y}(q)$ intended to approximate

$Y^*(q) = \{ d \in \mathcal{D} : d \models q \}.$

Construction follows a full-text-first evidence mining process. Annotators begin from a chosen target paper $d^*$, extract a set of natural-language constraints $C = \{c_i\}_{i=1}^m$, and refine them until the feasible set

$$\mathcal{M}(C) = \{ d \in \mathcal{D} : d \models C \}$$

shrinks to the singleton $Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$0. The paper describes this as a minimal sufficiency principle: the constraints must uniquely identify the target, but if any constraint is removed, alternative papers should appear. Some queries are then made intentionally unsatisfiable by perturbing a core constraint, thereby testing whether the agent can abstain rather than force a false positive (Xiong et al., 28 Apr 2026).

The clues are deliberately non-headline. Annotators mine them from subtle methodological details, proof or derivation details, local empirical observations, affiliation or organization relations, and citation relationships, while explicitly avoiding easy cues such as dataset name, central claim, or main contribution. This design makes multi-hop citation following and close full-text reading intrinsic rather than optional (Xiong et al., 28 Apr 2026).

Wide Research

Wide Research evaluates whether an agent can retrieve the full set of papers satisfying a technical specification. Here the answer space is inherently multi-valued:

$Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$1

The task is therefore a set completion problem rather than a single-hit identification problem. The dataset contains 400 queries, 3,692 total answers, an average of 9.23 valid answers/query, and answer cardinality in the range [2, 34] (Xiong et al., 28 Apr 2026).

Wide Research is constructed from academic entity graphs through a four-stage pipeline: domain-specific candidate sourcing, structural abstraction and query formulation, query refinement and initial verification, and iterative expansion with rigorous auditing. Candidate papers are checked by an ensemble of three frontier LLMs, and a paper is admitted only if those models unanimously agree that it satisfies the query. Human experts then audit the final sets and remove papers that marginally violate the constraints (Xiong et al., 28 Apr 2026).

A plausible implication is that Deep Research primarily stresses disambiguation under sparse clues, whereas Wide Research stresses boundary definition and completeness under conjunctive criteria. The paper’s own evaluation results support this split: agents often fail in Wide Research not because they cannot find any relevant papers, but because they do not know when the set is complete (Xiong et al., 28 Apr 2026).

3. Corpus, domain coverage, and construction protocol

The benchmark is built over a controlled scholarly corpus of more than 3 million arXiv papers, accessed through DeepXiv with full-text search, structured metadata, and agentic browsing tools (Xiong et al., 28 Apr 2026). This controlled environment is central to reproducibility. Unlike open-web settings, the benchmark does not depend on unstable snippets or continually changing page structures.

The benchmark spans eight core computer science domains. The paper describes the resulting coverage as broad and balanced, though it does not reduce the domains to a single aggregate difficulty index. Within Deep Research, about 90% of tasks have exactly one correct answer and about 10% are intentionally no-answer cases. These no-answer instances are not incidental; they operationalize abstention as a first-class capability rather than treating every query as guaranteed satisfiable (Xiong et al., 28 Apr 2026).

Construction is described as a full-text-first human–machine pipeline. For Deep Research, verification includes query rewriting to remove shortcuts, stress-testing with frontier models, human adversarial search, and uniqueness auditing. For Wide Research, verification includes large-scale candidate supplementation, LLM-based and human auditing, and multi-model consensus filtering (Xiong et al., 28 Apr 2026).

The benchmark’s design reflects a strong preference for paper-internal evidence over metadata-only retrieval. This suggests that the benchmark is intended not merely to measure corpus navigation, but to evaluate whether an agent can use scientific documents as compositional evidence objects whose decisive attributes may be distributed across sections and citation neighborhoods.

4. Evaluation environment and formal metrics

AutoResearchBench evaluates agents in a standardized ReAct-style environment over DeepXiv. The agent alternates among reasoning, searching, reading candidate results, and optionally terminating with a final answer. The appendix reports default settings including 30 max turns, a soft context budget of around $Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$2, 10 papers shown from the last search, default search list size $Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$3, planner sampling temperature 0.6, and maximum new tokens per completion 4096 (Xiong et al., 28 Apr 2026).

For Deep Research, correctness is strict exact match. The metric is defined as

$Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$4

This means that returning the wrong paper, returning extra papers, or failing to abstain when the gold set is empty all receive zero credit. Partial proximity to the target is not rewarded (Xiong et al., 28 Apr 2026).

For Wide Research, the paper explicitly rejects ranking-style evaluation in favor of Intersection over Union, because the objective is set completeness rather than top-$Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$5 ranking. The metric is

$Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$6

This formulation jointly penalizes missing valid papers and including invalid papers, thereby capturing the precision–recall tradeoff in a single set-valued score (Xiong et al., 28 Apr 2026).

A frequent misunderstanding is to read Wide Research as ordinary recall-heavy retrieval. The IoU choice makes the task stricter: exhaustive over-retrieval is not acceptable, because the benchmark evaluates whether an agent can infer the semantic boundary of the qualified set.

5. Baseline performance, tool comparisons, and failure modes

The paper evaluates a broad baseline set, including open-source models such as Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, DeepSeek-V3.2, MiniMax-M2.5, and Kimi-K2.5; closed-source models such as Qwen3-Max, Seed-2.0-Pro, Gemini-3-Flash, Gemini-3.1-Pro-Preview, GPT-5.4, Claude-Sonnet-4.6, and Claude-Opus-4.6; and end-to-end systems including Alphaxiv, GPT DeepResearch, and AI Studio Gemini-3.1-Pro (Xiong et al., 28 Apr 2026).

The headline result is that the benchmark is extraordinarily hard. On Deep Research, the best reported accuracy is 9.39% for Claude Opus 4.6, followed by 7.93% for Gemini-3.1-Pro-Preview and 7.44% for GPT-5.4. On Wide Research, the best reported IoU is 9.31% for Gemini-3.1-Pro-Preview, followed by 8.12% for GPT-5.4, 7.87% for Seed-2.0-Pro, and 7.70% for DeepSeek-V3.2. Many other strong baselines fall below 5%, and many open-source systems remain below 4% on Wide Research (Xiong et al., 28 Apr 2026).

The per-run statistics indicate that more search is not equivalent to better search. Claude Opus 4.6 averages 28.1 turns, Gemini-3.1-Pro-Preview averages 24.4 turns, and GPT-5.4 reaches competitive performance with only 6.1 turns, which the paper interprets as evidence that efficiency can matter more than long deliberation (Xiong et al., 28 Apr 2026).

The tool comparison between DeepXiv and an open-web Jina-based search backend further clarifies what the benchmark measures. Average Deep Research accuracy across matched models drops from 5.42% with DeepXiv to 3.97% with open-web search. The paper attributes this to the fact that many decisive clues are embedded in paper-internal evidence absent from titles, abstracts, and web snippets (Xiong et al., 28 Apr 2026).

The paper also reports that explicit Think reasoning has no consistent benefit: it sometimes helps slightly on Deep Research, is often worse on Wide Research, and consistently increases runtime and tool use. Test-time scaling helps more for Deep Research than Wide Research, suggesting that repeated sampling can partially repair brittle search trajectories but does not solve the more structural completeness problem in set recovery (Xiong et al., 28 Apr 2026).

Error analysis is unusually detailed. For Deep Research, the common errors are retrieval drift or semantic confusion, tool execution failures, evidence aggregation failures, and candidate ranking failures. For Wide Research, the common errors are constraint literalism-induced miss, premature search termination, precision-unconstrained candidate expansion, scientific knowledge coverage gap, and GT semantic boundary misalignment. The manual audit further finds that 96% of “extra” predictions were truly invalid, indicating that low precision is generally a genuine model failure rather than annotation incompleteness (Xiong et al., 28 Apr 2026).

6. Position within the benchmark landscape

AutoResearchBench occupies a distinct niche within the rapidly expanding literature on deep-research and agentic evaluation. DeepResearch Bench evaluates 100 PhD-level research tasks with report-quality and citation-grounding frameworks, emphasizing end-to-end web research and long-form report generation (Du et al., 13 Jun 2025). DeepResearch Bench II extends report evaluation through 132 grounded research tasks and 9,430 fine-grained binary rubrics, diagnosing failures in information recall, analysis, and presentation (Li et al., 13 Jan 2026). ResearcherBench instead targets 65 research questions across 35 AI subjects, focusing on whether deep AI research systems can act as research partners on frontier AI questions (Xu et al., 22 Jul 2025).

Other neighboring benchmarks stress different operating regimes. IDRBench evaluates interactive deep research, explicitly modeling user clarification and measuring both quality gains and interaction costs (Feng et al., 10 Jan 2026). DRBench moves the setting to enterprise environments, where tasks combine public web evidence with private organizational artifacts spread across applications such as email, chat, and shared storage (Abaskohi et al., 30 Sep 2025). RE-Bench departs from literature discovery entirely and measures open-ended ML research engineering against human experts in seven environments (Wijk et al., 2024). Even the web-focused Deep Research Bench: Evaluating AI Web Research Agents centers on open-web tasks with a frozen RetroSearch environment rather than full-text scholarly set completion (FutureSearch et al., 6 May 2025).

Against this backdrop, AutoResearchBench’s specificity is its defining feature. It is not a general report-writing benchmark, not an enterprise retrieval benchmark, not an interaction benchmark, and not an R&D engineering benchmark. It isolates scientific literature discovery as a standalone capability frontier, with strict attention to full-text evidence, unknown answer cardinality, abstention, and set completeness (Xiong et al., 28 Apr 2026).

This positioning also clarifies the meaning of its low absolute scores. They do not merely indicate weak browsing performance. Rather, they suggest that current frontier agents remain poor at sustained scientific search when the task requires exact discrimination among near neighbors, explicit reasoning over conjunctive paper-level constraints, and credible stopping behavior in a corpus-scale environment.