---
title: 'AutoResearchBench: Scientific Literature Discovery'
url: https://www.emergentmind.com/topics/autoresearchbench
type: topic
---

# AutoResearchBench: Scientific Literature Discovery

Searching arXiv for the benchmark paper and closely related deep-research benchmarks to ground the article in current literature.
AutoResearchBench is a benchmark for **autonomous scientific literature discovery**: the evaluation of whether AI agents can search a large scientific corpus, reason over full papers, and either identify a uniquely specified target paper or recover the complete set of papers satisfying a technical specification. Built over a controlled corpus of **more than 3 million arXiv papers** through the **DeepXiv** platform, it contains **1,000 total problems**—**600 Deep Research** and **400 Wide Research**—and is designed to be **research-oriented**, **literature-focused**, and **open-ended**. Its central empirical result is that the benchmark is extremely difficult: the best reported scores are **9.39% accuracy** on Deep Research and **9.31% IoU** on Wide Research, despite strong performance by frontier models on general agentic browsing benchmarks [2604.25256].

## 1. Research problem and benchmark rationale

AutoResearchBench targets a capability that is central to autonomous research but not well captured by generic web-search or browsing evaluations: **finding the right scientific literature under complex, multi-constraint, paper-level conditions**. The benchmark formalizes literature discovery as a setting in which the relevant evidence is often not recoverable from titles, abstracts, or snippets alone, but instead resides in **method sections, ablation tables, figure captions, appendices, citation chains, author affiliations, and experimental details** [2604.25256].

This focus distinguishes the benchmark from tasks that reduce research to short-answer retrieval or web-visible evidence. The benchmark’s stated motivation is that current benchmarks such as **GAIA**, **BrowseComp**, and related agentic web-browsing tasks do not adequately test **paper-level, research-oriented, open-ended literature discovery**. In this setting, an agent must reason about **correctness**, **completeness**, **abstention**, and **stopping criteria**, rather than merely ranking plausible documents [2604.25256].

The paper characterizes AutoResearchBench along three dimensions. It is **research-oriented** because the tasks require scientific understanding and technical reasoning; **literature-focused** because decisive evidence is usually embedded in full-text paper content; and **open-ended** because the answer set size is unknown and may be **one**, **zero**, or **many**. A common misconception is that the benchmark is simply a harder form of scholarly search. The design suggests a narrower and more demanding interpretation: it is a test of whether an agent can maintain hypothesis state over long search trajectories while discriminating between near-miss papers that differ only in subtle full-text constraints [2604.25256].

## 2. Task families: Deep Research and Wide Research

AutoResearchBench is organized around two complementary task types, each with a distinct answer-space structure and evaluation target [2604.25256].

| Task type | Gold answer structure | Primary metric |
|---|---|---|
| Deep Research | $|Y^*(q)| \in \{0,1\}$ | Exact-match accuracy |
| Wide Research | $Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}$ | Intersection over Union |

### Deep Research

Deep Research evaluates whether an agent can **precisely identify one target paper** from obfuscated, multi-step, full-text clues. The benchmark defines the gold set as satisfying

$$
|Y^*(q)| \in \{0,1\}.
$$

A query therefore has either **one unique correct paper** or **no correct paper**. Given a query $q$ and corpus $\mathcal{D}$, the agent predicts a set $\hat{Y}(q)$ intended to approximate

$$
Y^*(q) = \{ d \in \mathcal{D} : d \models q \}.
$$

Construction follows a **full-text-first evidence mining** process. Annotators begin from a chosen target paper $d^*$, extract a set of natural-language constraints $C = \{c_i\}_{i=1}^m$, and refine them until the feasible set

$$
\mathcal{M}(C) = \{ d \in \mathcal{D} : d \models C \}
$$

shrinks to the singleton $\{d^*\}$. The paper describes this as a **minimal sufficiency principle**: the constraints must uniquely identify the target, but if any constraint is removed, alternative papers should appear. Some queries are then made intentionally unsatisfiable by perturbing a core constraint, thereby testing whether the agent can **abstain** rather than force a false positive [2604.25256].

The clues are deliberately non-headline. Annotators mine them from **subtle methodological details**, **proof or derivation details**, **local empirical observations**, **affiliation or organization relations**, and **citation relationships**, while explicitly avoiding easy cues such as dataset name, central claim, or main contribution. This design makes multi-hop citation following and close full-text reading intrinsic rather than optional [2604.25256].

### Wide Research

Wide Research evaluates whether an agent can **retrieve the full set of papers** satisfying a technical specification. Here the answer space is inherently multi-valued:

$$
Y^*(q) = \{ d \in \mathcal{D} \mid d \models q \}.
$$

The task is therefore a **set completion** problem rather than a single-hit identification problem. The dataset contains **400 queries**, **3,692 total answers**, an average of **9.23 valid answers/query**, and answer cardinality in the range **[2, 34]** [2604.25256].

Wide Research is constructed from **academic entity graphs** through a four-stage pipeline: domain-specific candidate sourcing, structural abstraction and query formulation, query refinement and initial verification, and iterative expansion with rigorous auditing. Candidate papers are checked by an ensemble of **three frontier LLMs**, and a paper is admitted only if those models **unanimously agree** that it satisfies the query. Human experts then audit the final sets and remove papers that marginally violate the constraints [2604.25256].

A plausible implication is that Deep Research primarily stresses **disambiguation under sparse clues**, whereas Wide Research stresses **boundary definition and completeness under conjunctive criteria**. The paper’s own evaluation results support this split: agents often fail in Wide Research not because they cannot find any relevant papers, but because they do not know when the set is complete [2604.25256].

## 3. Corpus, domain coverage, and construction protocol

The benchmark is built over a controlled scholarly corpus of **more than 3 million arXiv papers**, accessed through **DeepXiv** with full-text search, structured metadata, and agentic browsing tools [2604.25256]. This controlled environment is central to reproducibility. Unlike open-web settings, the benchmark does not depend on unstable snippets or continually changing page structures.

The benchmark spans **eight core computer science domains**. The paper describes the resulting coverage as broad and balanced, though it does not reduce the domains to a single aggregate difficulty index. Within Deep Research, about **90%** of tasks have exactly **one** correct answer and about **10%** are intentionally **no-answer** cases. These no-answer instances are not incidental; they operationalize abstention as a first-class capability rather than treating every query as guaranteed satisfiable [2604.25256].

Construction is described as a **full-text-first human–machine pipeline**. For Deep Research, verification includes **query rewriting to remove shortcuts**, **stress-testing with frontier models**, **human adversarial search**, and **uniqueness auditing**. For Wide Research, verification includes **large-scale candidate supplementation**, **LLM-based and human auditing**, and **multi-model consensus filtering** [2604.25256].

The benchmark’s design reflects a strong preference for **paper-internal evidence** over metadata-only retrieval. This suggests that the benchmark is intended not merely to measure corpus navigation, but to evaluate whether an agent can use scientific documents as compositional evidence objects whose decisive attributes may be distributed across sections and citation neighborhoods.

## 4. Evaluation environment and formal metrics

AutoResearchBench evaluates agents in a standardized **ReAct-style** environment over DeepXiv. The agent alternates among **reasoning**, **searching**, **reading candidate results**, and optionally **terminating with a final answer**. The appendix reports default settings including **30 max turns**, a soft context budget of around **$1.1 \times 10^5$**, **10** papers shown from the last search, default search list size **$k = 10$**, planner sampling temperature **0.6**, and maximum new tokens per completion **4096** [2604.25256].

For Deep Research, correctness is strict **exact match**. The metric is defined as

$$
\text{Acc} = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \mathbb{I}(\hat{Y}(q) = Y^*(q)).
$$

This means that returning the wrong paper, returning extra papers, or failing to abstain when the gold set is empty all receive zero credit. Partial proximity to the target is not rewarded [2604.25256].

For Wide Research, the paper explicitly rejects ranking-style evaluation in favor of **Intersection over Union**, because the objective is set completeness rather than top-$k$ ranking. The metric is

$$
\text{IoU} = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \frac{|\hat{Y}(q) \cap Y^*(q)|}{|\hat{Y}(q) \cup Y^*(q)|}.
$$

This formulation jointly penalizes **missing valid papers** and **including invalid papers**, thereby capturing the precision–recall tradeoff in a single set-valued score [2604.25256].

A frequent misunderstanding is to read Wide Research as ordinary recall-heavy retrieval. The IoU choice makes the task stricter: exhaustive over-retrieval is not acceptable, because the benchmark evaluates whether an agent can infer the **semantic boundary** of the qualified set.

## 5. Baseline performance, tool comparisons, and failure modes

The paper evaluates a broad baseline set, including open-source models such as **Qwen3.5-35B-A3B**, **Qwen3.5-122B-A10B**, **Qwen3.5-397B-A17B**, **DeepSeek-V3.2**, **MiniMax-M2.5**, and **Kimi-K2.5**; closed-source models such as **Qwen3-Max**, **Seed-2.0-Pro**, **Gemini-3-Flash**, **Gemini-3.1-Pro-Preview**, **GPT-5.4**, **Claude-Sonnet-4.6**, and **Claude-Opus-4.6**; and end-to-end systems including **Alphaxiv**, **GPT DeepResearch**, and **AI Studio Gemini-3.1-Pro** [2604.25256].

The headline result is that the benchmark is extraordinarily hard. On Deep Research, the best reported accuracy is **9.39%** for **Claude Opus 4.6**, followed by **7.93%** for **Gemini-3.1-Pro-Preview** and **7.44%** for **GPT-5.4**. On Wide Research, the best reported IoU is **9.31%** for **Gemini-3.1-Pro-Preview**, followed by **8.12%** for **GPT-5.4**, **7.87%** for **Seed-2.0-Pro**, and **7.70%** for **DeepSeek-V3.2**. Many other strong baselines fall below **5%**, and many open-source systems remain below **4%** on Wide Research [2604.25256].

The per-run statistics indicate that more search is not equivalent to better search. **Claude Opus 4.6** averages **28.1 turns**, **Gemini-3.1-Pro-Preview** averages **24.4 turns**, and **GPT-5.4** reaches competitive performance with only **6.1 turns**, which the paper interprets as evidence that efficiency can matter more than long deliberation [2604.25256].

The tool comparison between **DeepXiv** and an **open-web Jina-based search** backend further clarifies what the benchmark measures. Average Deep Research accuracy across matched models drops from **5.42% with DeepXiv** to **3.97% with open-web search**. The paper attributes this to the fact that many decisive clues are embedded in **paper-internal evidence** absent from titles, abstracts, and web snippets [2604.25256].

The paper also reports that explicit **Think** reasoning has **no consistent benefit**: it sometimes helps slightly on Deep Research, is often worse on Wide Research, and consistently increases runtime and tool use. Test-time scaling helps more for Deep Research than Wide Research, suggesting that repeated sampling can partially repair brittle search trajectories but does not solve the more structural completeness problem in set recovery [2604.25256].

Error analysis is unusually detailed. For Deep Research, the common errors are **retrieval drift or semantic confusion**, **tool execution failures**, **evidence aggregation failures**, and **candidate ranking failures**. For Wide Research, the common errors are **constraint literalism-induced miss**, **premature search termination**, **precision-unconstrained candidate expansion**, **scientific knowledge coverage gap**, and **GT semantic boundary misalignment**. The manual audit further finds that **96% of “extra” predictions were truly invalid**, indicating that low precision is generally a genuine model failure rather than annotation incompleteness [2604.25256].

## 6. Position within the benchmark landscape

AutoResearchBench occupies a distinct niche within the rapidly expanding literature on deep-research and agentic evaluation. **DeepResearch Bench** evaluates **100 PhD-level research tasks** with report-quality and citation-grounding frameworks, emphasizing end-to-end web research and long-form report generation [2506.11763]. **DeepResearch Bench II** extends report evaluation through **132 grounded research tasks** and **9,430 fine-grained binary rubrics**, diagnosing failures in **information recall**, **analysis**, and **presentation** [2601.08536]. **ResearcherBench** instead targets **65 research questions** across **35 AI subjects**, focusing on whether deep AI research systems can act as research partners on frontier AI questions [2507.16280].

Other neighboring benchmarks stress different operating regimes. **IDRBench** evaluates **interactive deep research**, explicitly modeling user clarification and measuring both quality gains and interaction costs [2601.06676]. **DRBench** moves the setting to enterprise environments, where tasks combine public web evidence with private organizational artifacts spread across applications such as email, chat, and shared storage [2510.00172]. **RE-Bench** departs from literature discovery entirely and measures open-ended ML research engineering against human experts in seven environments [2411.15114]. Even the web-focused **Deep Research Bench: Evaluating AI Web Research Agents** centers on open-web tasks with a frozen **RetroSearch** environment rather than full-text scholarly set completion [2506.06287].

Against this backdrop, AutoResearchBench’s specificity is its defining feature. It is not a general report-writing benchmark, not an enterprise retrieval benchmark, not an interaction benchmark, and not an R&D engineering benchmark. It isolates **scientific literature discovery** as a standalone capability frontier, with strict attention to **full-text evidence**, **unknown answer cardinality**, **abstention**, and **set completeness** [2604.25256].

This positioning also clarifies the meaning of its low absolute scores. They do not merely indicate weak browsing performance. Rather, they suggest that current frontier agents remain poor at sustained scientific search when the task requires exact discrimination among near neighbors, explicit reasoning over conjunctive paper-level constraints, and credible stopping behavior in a corpus-scale environment.

Source: https://www.emergentmind.com/topics/autoresearchbench