PaperArena: Benchmark for Tool-Augmented LLM Agents

• PaperArena is an evaluation benchmark that enables tool-augmented, multi-modal reasoning over scientific literature using standardized execution protocols.
• It employs a modular platform (PaperArena-Hub) to coordinate multi-tool sequencing and cross-document evidence synthesis in AI research.
• The benchmark features 784 QA pairs drawn from AI papers, assessing both answer accuracy and tool-use efficiency in complex reasoning tasks.

PaperArena is an evaluation benchmark and execution platform for tool-augmented LLM agents operating over scientific literature. It is designed for realistic research questions that require integrating evidence across multiple papers, modalities, and external tools rather than answering isolated, tool-free questions within a single document. The benchmark contains 784 question-answer pairs grounded in real AI papers, and the accompanying PaperArena-Hub standardizes agent execution, tool interfaces, logging, and evaluation for both single-agent and multi-agent systems (Wang et al., 13 Oct 2025).

1. Concept and research setting

PaperArena addresses a specific gap in the evaluation of literature-facing agents: existing benchmarks are described as predominantly tool-free and single-paper, with narrow modality coverage and limited cross-document integration. The motivating claim is that real scientific work is neither text-only nor confined to one paper. Researchers routinely synthesize text, tables, figures, equations, and code across multiple documents and databases, while also verifying claims and computing quantitative summaries. In that setting, pure LLM reasoning is treated as insufficient; agents must parse PDFs, locate references, retrieve context, search external or structured sources, and execute code when numerical verification or replication is required (Wang et al., 13 Oct 2025).

The benchmark therefore defines literature reasoning as an explicitly tool-centric activity. Its questions are stated to be unanswerable by LLMs alone, compelling workflows that use external tools and evidence grounding. The required capabilities are organized along four axes: multi-step reasoning, multi-modal understanding, cross-document integration, and database interfacing. A representative question may require checking whether a claim made in one paper is actually supported by a cited prior work, extracting quantitative values from both papers, and computing the resulting difference before formulating an answer. This makes PaperArena closer to research practice than benchmarks such as PubMedQA, QASPER, and SPIQA, which are described as largely solvable without explicit tool invocation (Wang et al., 13 Oct 2025).

A frequent misunderstanding is to treat PaperArena as merely another question-answering dataset over papers. Its stated scope is broader: the benchmark evaluates not only answer correctness but also tool-usage efficiency and the reasoning process. PaperArena-Hub is therefore part of the benchmark’s core design rather than an auxiliary software release, because standardized execution is necessary if cross-paper reasoning and multi-tool orchestration are to be compared meaningfully across agent architectures (Wang et al., 13 Oct 2025).

2. Corpus construction and benchmark composition

The benchmark is built from a corpus of 14,435 open-access AI papers from recent 2025 OpenReview and Open Access sources. Each paper is mapped to a 20-dimensional binary vector reflecting influence, field, method, and evaluation type. Subset selection proceeds in two stages: K-Medoids selects 50 representative prototype papers, and Farthest Point Sampling adds 50 boundary papers to increase diversity. The resulting 100-paper subset is described as balancing representativeness and coverage (Wang et al., 13 Oct 2025).

Question generation follows a three-stage pipeline. First, MinerU parses PDFs into structured markdown with extracted text, tables, and figures. Second, Gemini 2.5 Pro produces draft QA pairs using a curated tool chain experience pool $\mathcal{E}$ intended to enforce multi-step, multimodal, and cross-paper or database reasoning. Third, semi-automated verification executes the drafted tool chain step by step, validates intermediate outputs, obfuscates questions to remove structural hints, standardizes answers, and creates distractors for multiple-choice items, after which human verification filters the set from 1,215 drafts to 784 final QA pairs (Wang et al., 13 Oct 2025).

The released benchmark spans multiple answer formats, difficulty levels, and capability types.

Category	Distribution
Question types	MC 126; CA 373; OA 285
Difficulty tiers	Easy 167; Medium 368; Hard 249
Capability distribution	Browsing 194; Coding 159; Multi-Modality 261; Multi-Steps 170

The “hard subset” is defined as questions requiring four or more sequential tool calls. This definition matters operationally because PaperArena treats tool sequencing as part of task difficulty rather than as an implementation detail. Grounding constraints also remain explicit throughout the dataset: answers may vary in style, but they must be unambiguous and supported by valid reasoning grounded in paper content and retrieved evidence, using only the tool suite defined in PaperArena-Hub. Execution is capped at 40 reasoning steps per question under standardized inference parameters (Wang et al., 13 Oct 2025).

3. PaperArena-Hub and the tool-augmented execution model

PaperArena-Hub is a modular, extensible, open-source execution environment built on smolagents. It supports both single-agent ReAct-style execution and a centralized multi-agent design in which a manager agent plans and delegates sub-tasks to worker agents specialized for parsing, retrieval, or coding. The platform includes planning, action execution, state updates, memory, reflection, logging, long-context management, and robust error handling (Wang et al., 13 Oct 2025).

The execution model is formalized as a sequence of tool-mediated state transitions,

$$s_0 \xrightarrow{t_1} s_1 \xrightarrow{t_2} \cdots \xrightarrow{t_n} s_n \to a, \qquad t_i \in \mathcal{T},$$

where the initial state $s_0$ is derived from the input papers and the question, each tool call updates the agent state, and the final answer $a$ is synthesized from the accumulated evidence. Standardized tool interfaces are intended to keep long tool chains executable and to reduce failures caused by inconsistent input-output formats (Wang et al., 13 Oct 2025).

The tool suite is partitioned into multimodal parsing, retrieval, and computation. Multimodal parsing consists of PDF Parser, Table Analyzer, and Figure Analyzer. Retrieval is handled by Context Retriever, Cross-Ref Searcher, Web Searcher, and Database Querier. Programmatic computation is delegated to Code Executor. The interfaces are specified concretely: PDF Parser takes pdf_path and returns markdown plus layout JSON; Table Analyzer takes table_id and a query and returns extracted textual or numeric data via structured HTML conversion; Figure Analyzer takes figure_id and a query and returns textual or numeric extraction using a multimodal LLM, with caption enrichment through GPT-4o for non-visual LLMs; Context Retriever takes query and top_k and returns semantically retrieved text chunks; Cross-Ref Searcher resolves a citation key to title, abstract, and URL via arXiv APIs plus LLM parsing; Web Searcher returns snippets and URLs via SerpAPI; Database Querier takes conference and status and returns papers with metadata through BM25 plus dense retrieval; Code Executor runs a code string in a sandboxed Python environment and returns stdout and stderr (Wang et al., 13 Oct 2025).

A representative PaperArena workflow illustrates the intended orchestration pattern. For a claim-verification task that compares Table 2 in one paper against cited results in another, the correct pipeline is specified as: PDF Parser on the first paper; Table Analyzer on the cited table; Cross-Ref Searcher to resolve the citation; Web Searcher or Database Querier to access the cited paper; Context Retriever to locate the relevant experimental section; Table Analyzer or Figure Analyzer on the second paper; Code Executor to compute the improvement; and final answer synthesis with grounded numerical evidence. This sequence embodies the benchmark’s central assumption that scientific reasoning is often a composition of parsing, retrieval, citation resolution, and computation rather than a single inference step (Wang et al., 13 Oct 2025).

4. Evaluation protocol, metrics, and judging

The reported evaluation covers nine leading LLMs in both single-agent and multi-agent configurations: Gemini 2.5 Pro, OpenAI o4-mini-high, Claude Sonnet 4, GPT-4.1, Claude 3.5 Sonnet, Qwen3-235B-Thinking, GLM-4.5, Qwen3-235B-Instruct, and Kimi-K2-Instruct. Standardized inference parameters are temperature 0.7, top_p 0.9, a maximum of 4096 output tokens, and a maximum of 40 reasoning steps. Human performance is measured on a 200-question subset, PaperArena-Human, using three CS Ph.D. candidates who were given 16 hours over two days and were allowed GPT-4o, web search, and Python (Wang et al., 13 Oct 2025).

Correctness is judged with GPT-4o in a binary LLM-as-a-Judge protocol, with human answers and a sample of LLM outputs manually verified. The reported agreement between the evaluator and humans on PaperArena-Human is approximately 98.5% for Gemini responses with Cohen’s $\kappa = 0.97$, and approximately 98.0% for Ph.D. responses with $\kappa = 0.93$, which is described as near-perfect alignment (Wang et al., 13 Oct 2025).

PaperArena evaluates both answer quality and process efficiency. The benchmark defines:

$$\text{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}[\hat{y}_i = y_i],$$

$$\text{AvgSteps} = \frac{1}{|Q|} \sum_{q \in Q} |\tau_q^{\text{exec}}|,$$

$$\text{AvgEff} = \frac{1}{|Q|} \sum_{q \in Q} \frac{|\tau_q^{\text{exec}} \cap \tau_q^{\text{gt}}|}{|\tau_q^{\text{exec}}|}, \qquad \text{AvgIneff} = 1 - \text{AvgEff},$$

and majority-vote test-time scaling through

$$\text{Acc}_k = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\left[\text{majority}(\{\hat{y}_{i,1},\ldots,\hat{y}_{i,k}\}) = y_i\right].$$

These definitions make a methodological distinction that is central to PaperArena: an agent may be correct while still inefficient, and high tool counts are not treated as evidence of sophistication. The use of ground-truth tool chains allows the benchmark to separate answer accuracy from endogenous planning quality and invocation fidelity, which is unusual among literature-oriented benchmarks (Wang et al., 13 Oct 2025).

5. Reported performance and observed failure modes

Experimental results indicate that the strongest reported multi-agent system, Gemini 2.5 Pro, reaches only 38.78% average accuracy on the full benchmark, and only 18.47% on the hard subset. Human experts achieve approximately 83.5% average accuracy, producing a large reported gap between current agentic systems and specialist human performance. Slow-thinking LLMs are reported to generally outperform fast-thinking variants, and closed-source models tend to outperform open-source models. Multi-agent designs improve both accuracy and efficiency on average, but they remain below the ceiling observed when a single agent is given a perfect tool chain in a planning-ceiling experiment (Wang et al., 13 Oct 2025).

Efficiency measurements sharpen this picture. Gemini 2.5 Pro in the single-agent setting reaches approximately 41.20% average efficiency with 9.81 average reasoning steps, whereas the multi-agent version reaches approximately 45.39% average efficiency with 8.58 steps. Human experts reach approximately 75.48% average efficiency with 6.52 steps. The performance gap is therefore not only a matter of correctness but also of route selection through the tool space (Wang et al., 13 Oct 2025).

Tool-use analysis identifies a systematic bias toward general-purpose tools. Agents preferentially overuse Web Searcher and Code Executor, while specialized tools such as PDF Parser are underused relative to task structure. Practical Code Executor calls exceed the theoretical requirement by more than $$s_0 \xrightarrow{t_1} s_1 \xrightarrow{t_2} \cdots \xrightarrow{t_n} s_n \to a, \qquad t_i \in \mathcal{T},$$0, and Web Searcher calls exceed it by more than $$s_0 \xrightarrow{t_1} s_1 \xrightarrow{t_2} \cdots \xrightarrow{t_n} s_n \to a, \qquad t_i \in \mathcal{T},$$1; PDF Parser calls, by contrast, align closely with theory for deterministic tasks. This suggests that many current agents default to broad fallback tools instead of constructing narrow, evidence-preserving pipelines (Wang et al., 13 Oct 2025).

The reported single-agent failure modes are parameter error at 21.8–39.0%, logical error at 22.9–30.9%, redundant action at 18.8–39.1%, and tool failure at 13.8–16.3%. Tool failure is therefore the smallest of the listed categories, and the paper explicitly notes that the tools themselves are generally reliable. This is important because it contradicts a common objection that poor benchmark performance is primarily caused by unstable infrastructure; the results attribute most failures to invocation quality, planning, and redundancy rather than to tool malfunction (Wang et al., 13 Oct 2025).

Pass@k evaluation shows that accuracy improves as $$s_0 \xrightarrow{t_1} s_1 \xrightarrow{t_2} \cdots \xrightarrow{t_n} s_n \to a, \qquad t_i \in \mathcal{T},$$2 increases, but gains diminish at larger $$s_0 \xrightarrow{t_1} s_1 \xrightarrow{t_2} \cdots \xrightarrow{t_n} s_n \to a, \qquad t_i \in \mathcal{T},$$3 and costs grow nearly linearly under naive parallelization. PaperArena therefore treats test-time scaling as a limited remedy rather than a substitute for better planning and tool selection (Wang et al., 13 Oct 2025).

6. Position within the evaluation landscape, implementation, and future directions

PaperArena is situated beyond literature QA datasets such as PubMedQA, QASPER, SPIQA, CharXiv, emrQA, QASA, and ArgSciChat because it combines cross-document synthesis, multimodal grounding, database interfacing, and explicit multi-tool workflows. Relative to more general agent benchmarks such as GAIA and BrowseBench variants, and literature-oriented systems such as PaperQA and STELLA, its defining target is scientific literature-specific tool orchestration together with structured evaluation of tool chain efficiency (Wang et al., 13 Oct 2025).

The name also places it within a broader family of “arena” evaluation systems, but its emphasis is distinct. Inclusion Arena is a live, app-integrated platform for pairwise preference ranking of foundation models under real user interactions, while CATArena evaluates agent learning ability through iterative tournament competitions over open-ended games. PaperArena instead centers on grounded reasoning over scientific documents and evidence-bearing tool chains, rather than user preference aggregation or tournament-based peer learning (Wang et al., 15 Aug 2025, Fu et al., 30 Oct 2025).

The implementation is openly released through a public repository, with code and data available at https://github.com/Melmaphother/PaperArena. The dataset includes 784 QA triplets with structured metadata and ground-truth tool chains, plus the 200-question PaperArena-Human subset. PDFs are parsed to markdown, and QA data are released as structured JSON objects containing question, answer, metadata, required tool_chain, and rationale. The release uses an open, non-commercial academic license with proper attribution required and with explicit attention to underlying paper copyrights and public open-access sourcing (Wang et al., 13 Oct 2025).

Practical guidance in the release emphasizes modular agent design, explicit tool-chain planning, memory over prior tool outputs, preference for specialized tools where appropriate, parameter validation to reduce invocation errors, and reflection or self-correction loops. For evaluation, the recommended reporting set is Acc, AvgSteps, AvgEff, and AvgIneff under standardized parameters, with optional pass@k analysis and dedicated error attribution across parameter, logical, redundancy, and tool-failure categories. The hard subset is recommended as a stress test for four-step-or-more planning (Wang et al., 13 Oct 2025).

Several limitations are stated directly. Current coverage is restricted to recent AI papers rather than broader fields such as biology, chemistry, or clinical medicine. LLM-as-a-Judge, despite high agreement with humans, may still introduce edge-case biases. Benchmark execution is expensive because long tool chains and repeated calls increase both latency and cost. Planned extensions include broader domains and modalities, improved question generation and obfuscation, additional metrics for plan quality, tool rationale, interpretability, and cost-aware efficiency, and agent designs with better memory, error correction, and heterogeneous multi-LLM coordination (Wang et al., 13 Oct 2025).