PaSaMaster Benchmark for Agentic Literature Retrieval

• The paper introduces PaSaMaster Benchmark as a novel framework for evaluating literature retrieval via target-set recovery, emphasizing full constraint satisfaction and intent fidelity.
• PaSaMaster employs a multi-stage, expert-driven pipeline that combines question generation, multi-channel retrieval, and checklist-based expert annotation to verify candidate papers.
• The benchmark demonstrates significant performance gains with a 15.6X F1 improvement over conventional systems, eliminating hallucinations and reducing computational costs.

Searching arXiv for the benchmark paper and related context. PaSaMaster Benchmark, also described as PaSaMaster-Bench, is a benchmark for complex scientific literature retrieval introduced alongside the PaSaMaster system in "Towards Self-Evolving Agentic Literature Retrieval" (Du et al., 14 May 2026). It is designed to evaluate whether a retrieval system can understand complex, realistic, natural-language research intents and recover the full set of real papers satisfying all stated constraints, rather than merely returning vaguely related documents for short keyword queries. In the paper’s framing, the benchmark is the empirical backbone for a shift from one-shot query–document matching toward "self-evolving agentic retrieval," in which ranked evidence is used to reveal gaps, refine intent understanding, and guide follow-up searches (Du et al., 14 May 2026).

1. Conceptual definition and motivation

PaSaMaster-Bench defines scientific literature retrieval as a target-set recovery problem over complex research intents. Each task presents a detailed natural-language query that may combine multiple constraints, including topic, method, application setting, benchmark dataset, publication restriction, time condition, or exclusions. A system must autonomously search for papers, verify them, and return a ranked list $P_{\text{agent}} = [P_1, P_2, \ldots, P_k]$, which is then compared against an expert-annotated ground-truth target set $P^*$ (Du et al., 14 May 2026).

The motivation for the benchmark is explicit. The paper argues that existing literature-retrieval evaluations are too weak for the problem under study. Traditional systems such as Google Scholar preserve source authenticity because they return indexed papers, but they compress rich search intents into keywords. Semantic retrievers improve matching, but still treat retrieval as static query–document matching. Generative LLMs can understand richer language, but may hallucinate papers or citation metadata. Fixed-pipeline agents reduce hallucination by using tools, but still lock the intent interpretation at the beginning of the search. PaSaMaster-Bench is therefore constructed so that success depends on constraint reasoning, intent fidelity, and source verifiability rather than shallow topical similarity (Du et al., 14 May 2026).

A common misconception is to treat the benchmark as a conventional relevance dataset for broad semantic similarity. The paper explicitly rejects that interpretation. Relevance is conjunctive and checklist-based: a paper is correct only if it satisfies the full query intent, not merely some keywords or an approximate theme. This suggests that the benchmark is aimed at evaluating retrieval under realistic research workflows, where omitted constraints and partially matching papers are substantive errors rather than benign approximations.

2. Task formulation and relevance criterion

The task format is simple in interface but strict in semantics. The input is a complex natural-language query. During annotation, each query is paired with a checklist of objective, verifiable constraints that operationalizes relevance. The system-facing output is a ranked list of papers, evaluated up to the cutoff $K=20$ (Du et al., 14 May 2026).

The key design distinction is that membership in $P^*$ requires satisfying all required checkpoints in the query’s checklist. The benchmark does not use graded relatedness labels. It repeatedly emphasizes an all-constraints criterion, which is why its precision and F1 are intended to reflect true intent understanding rather than broad semantic overlap (Du et al., 14 May 2026).

The evaluation protocol centers on top-20 retrieval quality. The reported metrics are:

Metric	Interpretation
Recall@20	How completely the system recovers the target papers implied by the query
Precision@20	Whether retrieved papers truly satisfy the expert-defined full intent
F1-score@20	Harmonic mean of Recall@20 and Precision@20
NDCG@20	Ranking quality, rewarding earlier placement of true target papers

The benchmark also measures two additional dimensions aligned with the paper’s thesis. Hallucination rate is defined as the proportion of returned papers that cannot be verified as real scientific sources. Token usage / cost is computed from total input and output tokens consumed during inference, priced according to the corresponding model rates; the paper notes that Gemini 3.1 cost is estimated using a 20-question sample (Du et al., 14 May 2026).

The paper does not report a train/dev/test split in the provided text. It presents the benchmark as 244 tasks used for evaluation, but gives no partitioning details. That absence is part of the benchmark’s current documented scope.

3. Coverage across disciplines

PaSaMaster-Bench contains 244 independent literature discovery tasks spanning 38 scientific disciplines (Du et al., 14 May 2026). It is deliberately multidisciplinary rather than CS-only, and the paper uses this breadth to argue that the benchmark stress-tests retrieval under disciplinary heterogeneity.

Figure 1 groups the tasks into five broad areas:

Area	Questions
AI and Computing	76 (31.1%)
Engineering and Technology	59 (24.2%)
Medicine and Life Sciences	66 (27.0%)
Basic Sciences	25 (10.2%)
Others	18 (7.4%)

The same figure gives examples of subareas and counts, including Artificial Intelligence (48), Computer Science (22), Materials Science and Engineering (25), Clinical Medicine (37), Chemistry (7), Earth Science (6), Physics (10), Mathematics (2), Management (4), Education (2), Linguistics (2), and Psychology (2). In the appendix, the paper further describes a broader construction taxonomy with 19 top-level disciplines and 97 fine-grained topics, averaging 5.1 fine-grained topics per discipline. The listed fields include geoscience, physics and astrophysics, mathematics, agroforestry science, materials science, computer science, artificial intelligence, chemistry, engineering, biology, medicine, law, psychology, pedagogy, economics, management, and humanities (Du et al., 14 May 2026).

This disciplinary spread matters because the benchmark is meant to test whether retrieval systems can handle heterogeneous domain conventions, terminology, and constraint structures. A plausible implication is that the benchmark is not only measuring semantic matching quality, but also robustness to differences in how scientific intents are expressed across fields.

4. Construction pipeline and annotation philosophy

The benchmark is built through an expert-driven curation pipeline, described both as a "two-stage expert-driven pipeline" and, in Figure 1, as a three-part process (Du et al., 14 May 2026).

First, in question generation, domain experts formulate complex natural-language literature search queries based on authentic research bottlenecks. For each query, they also provide a constraint checklist decomposing the query into objective, verifiable criteria.

Second, in multi-channel retrieval, each query is issued to several strong retrieval or generation systems to create a broad candidate pool. The paper states that these systems include web-enabled frontier LLMs such as GPT-5.2 and Gemini 3.1 Pro, PaSaMaster’s native search engine, and traditional web search. Retrieved papers are verified against the corpus, deduplicated, and merged into a unified candidate set.

Third, in expert annotation, domain experts inspect candidate papers against the predefined checklist and assign checkpoint-level judgments. Only papers satisfying all required checkpoints are included in the ground-truth target set $P^*$ (Du et al., 14 May 2026).

This candidate-pool strategy is methodologically important because the gold set is not constructed from the outputs of one retrieval method alone. The paper presents this as an attempt to make recall of the target set more realistic by aggregating candidates from multiple strong channels before expert adjudication.

The annotation philosophy is governed by four stated principles:

• Intent fidelity: tasks should reflect real research intents, not synthetic keyword queries.
• Bounded recall: the target paper set should be sufficiently well-defined for reliable expert annotation.
• Authentic complexity: the query should require nontrivial interpretation of multiple constraints.
• Verifiable correctness: every gold paper must be justified by checklist-based evidence.

These principles clarify that the benchmark is not evaluating a soft notion of topical relatedness. It is evaluating whether a retrieval system can reconstruct an expert-defined target set under explicit, verifiable constraints.

5. Relation to the PaSaMaster retrieval paradigm

PaSaMaster-Bench is tightly coupled to the retrieval paradigm proposed in the paper. PaSaMaster itself is implemented as a retrieval-and-ranking system over a verified scientific repository rather than as a citation generator, and the benchmark is designed to reward precisely that distinction (Du et al., 14 May 2026).

The system operates over a customized scientific corpus of over 160 million papers, reorganized as a three-tier repository

$$D = \{D_{\text{meta}}, D_{\text{abs}}, D_{\text{chunk}}\},$$

where $D_{\text{meta}}$ stores metadata, $D_{\text{abs}}$ supports abstract-level coarse filtering, and $D_{\text{chunk}}$ contains passage-level evidence segments from full texts (Du et al., 14 May 2026).

Its planning–retrieval split is formalized as

$(S, C) = PLAN(q, T, \pi_{Nav}), \tag{1}$

$P^*$0

$P^*$1

$P^*$2

where $P^*$3 is the retrieval strategy, $P^*$4 is the query-specific checklist, and $P^*$5 parallel Librarian agents execute retrieval and verification (Du et al., 14 May 2026).

The "self-evolving" component is captured by iterative reflection: $P^*$6 Ranked evidence from one round updates both the strategy and the verification checklist for the next round. The benchmark is explicitly designed to favor systems that can revise their interpretation of complex intents over time rather than issuing a one-shot static query (Du et al., 14 May 2026).

The evidence-grounded scorer also links the benchmark’s relevance notion to localized textual support. For each checklist item $P^*$7, supporting chunks are localized by

$P^*$8

checkpoint scores $P^*$9 are averaged into

$K=20$0

and the final normalized relevance score is

$K=20$1

where $K=20$2 is the scorer’s calibrated probability for the overall relevance judgment (Du et al., 14 May 2026).

This architecture explains how the benchmark’s source-authenticity requirement is operationalized. Returned papers come from verified corpora, and relevance judgments are tied to actual evidence segments rather than citation generation from parametric memory.

6. Baselines, results, and interpretive significance

The benchmark compares systems spanning what the paper presents as a retrieval-paradigm ladder: lexical retrieval (Google Scholar), semantic retrieval (OpenScholar, Bohrium Science Navigator), generative LLMs with search/visit tools (DeepSeek-v3.2, Kimi-K2.5, MiniMax-M2.7, GLM-5, Gemini-3.1-pro, GPT-5.2), fixed-pipeline agentic retrieval (Google Scholar Labs), and self-evolving agentic retrieval (PaSaMaster) (Du et al., 14 May 2026).

For the generative LLM baselines, the setting is stronger than closed-book generation. The paper states that these models are equipped with Search and Visit tools and prompted to perform ReAct-style literature discovery before returning a final paper list. The benchmark therefore tests hallucination even under web-enabled, tool-augmented conditions (Du et al., 14 May 2026).

The main results table reports the following values:

System	NDCG@20 / Recall@20 / Precision@20 / F1@20
PaSaMaster	37.93 / 31.84 / 22.19 / 21.69
Google Scholar	2.07 / 1.69 / 1.48 / 1.39
OpenScholar	14.61 / 11.68 / 8.52 / 7.92
Bohrium Science Navigator	22.39 / 19.37 / 12.50 / 12.26
Google Scholar Labs	30.54 / 29.01 / 18.79 / 18.87
GPT-5.2	31.59 / 25.32 / 16.82 / 16.69

The paper’s headline 15.6X F1 improvement is the ratio between PaSaMaster and Google Scholar: $K=20$3 It also reports that, relative to strong non-PaSaMaster baselines, F1 improves from 18.18 to 21.69 versus GLM-5, a 19.3% improvement, and from 18.87 to 21.69 versus Google Scholar Labs, a 14.9% improvement (Du et al., 14 May 2026).

Hallucination is a central result. The reported hallucination rates are 20.57% for DeepSeek-v3.2, 35.67% for Kimi-K2.5, 37.79% for MiniMax-M2.7, 29.07% for GLM-5, 32.41% for Gemini-3.1-pro, and 11.80% for GPT-5.2, while PaSaMaster reports 0% because it ranks retrieved papers from verified corpora rather than generating citations (Du et al., 14 May 2026). Table 3 further decomposes hallucinations by metadata field. For example, GPT-5.2 has title 0.77%, author 1.73%, date 7.88%, link 0.93%, all 11.80%; MiniMax-M2.7 has 8.74%, 16.56%, 30.00%, 10.29%, all 37.79%. This field-level analysis is significant because unverifiability may arise from corrupted metadata even when a title-like string appears plausible.

Cost is treated as a first-class evaluation dimension. PaSaMaster is reported at \$K=20$46.06 for GPT-5.2. The paper states that PaSaMaster outperforms GPT-5.2 in F1-score by 30.0% while using only about 1% of the computational cost, based on

$K=20$5

and

$K=20$6

These results are used to support the paper’s claim that separating planning from retrieval can improve both accuracy and efficiency (Du et al., 14 May 2026).

The paper also states that Figure 2 shows consistent or comparable per-discipline F1 performance across AI and Computing, Engineering and Technology, Medicine and Life Sciences, Fundamental Sciences, and Humanities/Social Sciences/Others, although exact per-discipline values are not tabulated in the provided text. The benchmark’s breadth is therefore used to argue that the observed gains are not confined to a single subject area.

7. Limitations, release status, and place in retrieval evaluation

PaSaMaster-Bench is presented as expert-curated, multidisciplinary, checklist-annotated, and constructed from candidate pools aggregated from multiple retrieval channels. The public repository link provided in the paper is: https://github.com/sjtu-sai-agents/PaSaMaster (Du et al., 14 May 2026).

At the same time, the provided text leaves several benchmark-governance details unspecified. It does not provide train/dev/test splits, inter-annotator agreement, or exact benchmark file schemas. The paper does, however, provide what it calls the core ingredients necessary to understand and reproduce the benchmark’s intent: authentic expert-written queries, explicit verifiable criteria, omni-channel candidate mining, and expert admission of papers into gold sets only when all checklist items are satisfied (Du et al., 14 May 2026).

In the broader literature-retrieval landscape, the benchmark is positioned against three alternative paradigms. Lexical and semantic systems issue a single retrieval interpretation of the query. Generative LLMs often synthesize a one-shot answer list, even when aided by tools. Fixed-pipeline agents retrieve, read, and answer, but typically do not update their internal interpretation of the intent based on ranked evidence. PaSaMaster-Bench is designed to stress the failure modes of these paradigms by using realistic, multi-constraint, natural-language queries whose correct target set is recoverable only when a system can identify missing constraints, under-covered subcases, ambiguity, exclusions, and evidence-backed checklist satisfaction (Du et al., 14 May 2026).

For that reason, the benchmark is presented not merely as an evaluation dataset but as an operationalization of a theoretical claim: literature retrieval should be modeled as a self-evolving intent–paper ranking process. Its empirical role in the paper is to show that evaluating literature retrieval on short keyword queries or loosely related documents is insufficient when the task is faithful recovery of real papers satisfying a full research intent (Du et al., 14 May 2026).