Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

ScholarCatalyst Benchmark

Updated 3 October 2026
  • ScholarCatalyst are a set of benchmarks and datasets that identify papers whose ideas advance early-stage research by catalytic relevance, rather than by topical or semantic similarity.
  • ScholarCatalyst contain over 4,845 queries and 190,896 papers, focusing on computer science domains---the tasks require identifying papers relevant to specific incomplete research problems defined by preliminary research questions and unconstrained solution options.
  • Evolving enhancement of supervisor machine learning models remains continuing benchmarks indicators & measures to increase the reality predictive power & few incidents of application.

ScholarCatalyst is a benchmark for retrieving papers whose ideas could advance an early-stage research project, rather than merely papers with topical or semantic similarity. It uses author-provided judgments from 184 researchers associated with 207 recent computer-science papers, constructing 894 research queries and 4,845 validated positive query–paper pairs. The benchmark evaluates whether retrieval systems can identify methods, limitations, phenomena, conceptual framings, evidence, and other forms of intellectual contribution that researchers considered capable of moving a project forward (Yang et al., 10 Feb 2026).

1. Scope and conceptual foundations

ScholarCatalyst distinguishes topical relevance from catalytic relevance. Topical relevance concerns shared terminology, entities, tasks, methods, or subject areas. Catalytic relevance concerns whether a paper offers an idea that could help formulate, redirect, investigate, or solve a research problem. A catalytic paper may provide an adaptable method, expose a limitation, supply empirical evidence, suggest a conceptual reframing, or connect apparently separate fields.

The benchmark is motivated by the observation that conventional literature-search systems and citation graphs incompletely represent the intellectual origins of research. A paper may influence a project without being cited, while a cited paper may be topically related but not intellectually important to the project. ScholarCatalyst therefore treats inspiration as a latent relation between an initial research question and a prior paper, inferred from expert author judgments rather than citation status alone.

Each benchmark query describes the state of knowledge at the beginning of a project, the unresolved problem, and the open question confronting the researchers, while withholding the eventual solution. The system must rank papers that were available by the time the project began. The source paper, completed solution, and source bibliography are withheld in the principal query-only setting.

The benchmark operationalizes a component of scientific “research taste”: identifying which prior ideas matter for a particular research trajectory, including connections across subfields. This does not imply that catalytic relevance is an intrinsic, universal property of a paper. The same paper may serve as a method source in one project, evidence in another, a limitation motivating a third, or a hard negative in a fourth.

2. Dataset construction and temporal design

The dataset contains 184 researchers, 207 source papers published during 2025–2026, and 894 queries. The source projects are concentrated in artificial intelligence and adjacent computer-science fields.

Source-paper domain Papers Percentage
Machine learning (cs.LG) 66 31.9%
Computer vision (cs.CV) 46 22.2%
Computation and language (cs.CL) 38 18.4%
Artificial intelligence (cs.AI) 23 11.1%
Robotics (cs.RO) 20 9.7%
Other 14 6.8%

More than 100 of the 207 source papers had received an oral, spotlight, or award distinction at a major AI conference. Recruitment targeted first, co-first, and corresponding authors through conference-paper lists, direct outreach, and an open call.

Each source project contributes a CoreQ and one or more SubQs. A CoreQ is a broad question concerning which prior ideas could help address the project’s central problem. A SubQ focuses on a distinct research direction or subfield that informed the project.

Query type Queries Average positives Cited / uncited positives Average hard negatives Average length
CoreQ 207 3.19 95.5% / 4.5% 10.01 131.8 tokens
SubQ 687 6.09 56.4% / 43.6% 6.88 63.5 tokens

The retrieval corpus contains approximately 191,000 papers, with a detailed count of 190,896. It includes resolved references from source papers and approximately 181,000 additional arXiv papers from 2020–2024, principally from cs.CV, cs.LG, cs.CL, cs.RO, and cs.AI. Of the corpus, 189,296 papers, or 99.2%, come directly from arXiv; 1,600 additional records come from Semantic Scholar or OpenAlex when cited works were unavailable on arXiv.

For a source paper SS, the admissible search set is defined by publication date:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.

This temporal constraint prevents systems from exploiting later descriptions of the source paper or its solution. Web search is disabled in the principal setting, and the principal agent backbones have knowledge cutoffs before the source papers. The median searchable corpus contains approximately 190,574 papers per query.

The corpus is distributed as follows:

Corpus category Papers Percentage
cs.CV 64,919 34.0%
cs.LG 57,151 29.9%
cs.CL 33,255 17.4%
cs.RO 18,922 9.9%
cs.AI 10,933 5.7%
Other arXiv categories 4,116 2.2%
Semantic Scholar/OpenAlex 1,600 0.8%

The benchmark contains 4,845 author-validated positive query–paper pairs. CoreQs contain fewer positives and are longer and broader than SubQs. SubQs contain a substantially larger fraction of uncited positives, making them particularly important for evaluating discovery beyond bibliographic expansion.

3. Annotation, candidate generation, and inspiration types

ScholarCatalyst uses an automated pre-annotation pipeline followed by author validation. Starting from an arXiv identifier, the pipeline obtains metadata and source files through the arXiv API, parses the source paper, resolves bibliography entries to canonical records, and reconstructs CoreQs, SubQs, candidate inspirations, and rationales using Gemini 3.1 Pro.

Additional potentially uncited candidates are retrieved with BM25 and Qwen3-Embedding-8B. The candidate pool is reranked with Gemini 3.6 Flash, and approximately 10 candidates per query are retained for author review.

The pre-annotation pipeline is useful but insufficient. Reconstructed CoreQs were rated accurate or mostly accurate in 98.1% of cases, and SubQs in 95.9%. Approximately 85.1% of originally cited SubQ candidates were retained as positives, while 35.1% of pooled non-cited candidates were judged genuine inspirations. For automatically nominated CoreQ key inspirations, precision was 58.2%, recall was 26.0%, and the complete nominated set exactly matched the author’s set in only 9.2% of cases.

Authors can validate, rewrite, or replace the generated questions and candidate lists. For each candidate paper, they indicate whether it offers an insightful idea for pursuing the query and provide a rationale. Positive and negative sets can be represented as:

Dq+={papers judged by the author to offer useful inspiration for q},\mathcal{D}_q^+=\{\text{papers judged by the author to offer useful inspiration for }q\},

Dq−={reviewed papers that were topically related but not inspirational}.\mathcal{D}_q^-=\{\text{reviewed papers that were topically related but not inspirational}\}.

A manual consistency review found that 764 of 894 queries, or 85.5%, passed unchanged. The remaining queries were lightly reworded to remove wording that revealed the eventual solution or represented the research as completed.

Hard negatives are topically related papers that were either cited by the source paper or ranked highly by a retrieval system. They are not randomly selected irrelevant documents. This construction requires systems to distinguish intellectually useful work from plausible but non-catalytic neighboring research.

The benchmark identifies nine multi-label inspiration types:

  1. Method: a method, component, formulation, or principle is adopted or adapted.
  2. Generalization: an idea is extended to a new task, modality, domain, or setting.
  3. Limitation: a limitation or failure motivates the new work.
  4. Framework: an overall framework or system is reused.
  5. Phenomenon: an observed empirical effect motivates investigation.
  6. Reframing: a new conceptual perspective is supplied.
  7. Motivation: the paper motivates the problem or direction.
  8. Evidence: empirical evidence supports a research direction.
  9. The displayed taxonomy is described as containing nine types, with motivation and evidence completing the list in the supplied description.

Rationales receive 1.69 labels on average. More than one type applies in 55.8% of cases, and at least three labels apply in 12.7% of cases. The most frequent roles include adapted methods, supporting evidence, generalization, and limitations that motivate new approaches.

Approximately 43.6% of SubQ positives were not cited by the source paper. Of these uncited positives, 67.8% shared no references with the source paper. Among cited positives, only 46.2% of CoreQ positives and 34.0% of SubQ positives received a “uses” or “extends” citation-intent label. In addition, 49.5% of credited papers came from outside the source project’s subject area.

These results demonstrate that citation status is an incomplete proxy for scientific inspiration. The benchmark’s positive labels instead encode author-level judgments about transferable intellectual utility.

4. Retrieval task, metrics, and baselines

For each query qq associated with source paper SS, a system must rank the temporally admissible corpus:

rank⁡(q,P<S),\operatorname{rank}(q,\mathcal{P}_{<S}),

so that the author-validated positive set appears near the top.

Binary relevance is defined as:

rel⁡(q,p)={1,p∈Dq+, 0,p∉Dq+.\operatorname{rel}(q,p)= \begin{cases} 1, & p\in\mathcal{D}_q^+,\ 0, & p\notin\mathcal{D}_q^+. \end{cases}

For a ranked list Lq=(p1,…,pk)L_q=(p_1,\ldots,p_k), Recall@kk is:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.0

Aggregate Recall@P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.1 is the average across queries. Thus, Recall@20 measures the fraction of an author’s positive papers appearing among the first 20 retrieved papers, rather than the fraction of queries for which at least one positive is retrieved.

The benchmark also reports normalized discounted cumulative gain:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.2

and:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.3

Reported metrics include nDCG@20 and Recall@5, Recall@20, and Recall@100.

The evaluated retrieval baselines include BM25, LateOn 0.1B, SPECTER2, SciNCL, OpenScholar, Qwen3-Embedding-4B, Qwen3-Embedding-8B, Gemini Embedding 2, GTE-Qwen2-7B-Instruct, ReasonEmbed, Inf-Retriever-v1-Pro, NV-Embed-Reasoning-3B, and DIVER-Retriever-4B.

Embedding-based ranking can be represented conceptually as:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.4

where P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.5 and P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.6 produce query and paper representations. Papers are ranked by decreasing similarity. BM25 performs lexical matching, whereas LateOn aggregates token-level similarities.

The Qwen3 retrieval instruction is:

Given a scientific research question, retrieve papers that could inspire or contribute to answering the question, including relevant background knowledge, methods, and ideas.

The inclusion of this instruction emphasizes that the target relation is not ordinary topical similarity. It explicitly requests papers that could contribute ideas, methods, background, or conceptual resources.

For agentic tool-calling search, each candidate is scored from 0 to 10 according to how much its ideas could help pursue the query. If P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.7 is the candidate pool and P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.8 is the model’s usefulness score, the approximate ranking is:

P<S={p∈P:date⁡(p)≤date⁡(S)}.\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.9

Tournament reranking divides candidates into batches of 20 and advances the top four from each batch.

5. Agentic search and reported findings

The benchmark compares embedding retrieval with agentic systems that use retrieval as a tool. The principal reported result is that agentic search does not outperform embedding retrieval: the agent achieves Recall@20 of 0.42, compared with 0.48 for embedding retrieval. An agent built on Claude Fable 5.1 reaches only 0.51 Recall@20, despite the possibility that the model may have encountered completed source papers during pretraining.

This finding suggests that tool use and iterative search do not automatically produce expert-level retrieval. An agent may call the same retriever repeatedly without acquiring the latent judgment required to distinguish catalytic relevance from topical relevance. A plausible implication is that agentic systems require specialized training on expert rationales, hard-negative discrimination, query reformulation, and cross-domain analogy rather than merely access to stronger general-purpose LLMs.

The reported result also separates search orchestration from relevance judgment. An agent can plan searches, invoke tools, read candidate papers, and rerank results, yet still fail to identify the papers that an expert regards as intellectually catalytic. The missing capability is not necessarily retrieval coverage alone; it may involve recognizing why a paper’s idea transfers to a partially specified research problem.

The benchmark’s evaluation should not be interpreted as showing that agents are universally inferior to embedding systems. The comparison is specific to the ScholarCatalyst task, corpus, temporal restrictions, models, and reported configurations. It does show that agentic search provides no automatic advantage under these conditions.

6. Scientific significance, limitations, and future directions

ScholarCatalyst contributes an author-validated evaluation framework for a form of literature retrieval that is difficult to capture with citations, keywords, or ordinary semantic similarity. Its principal innovations are the use of early-stage research questions, author-supplied rationales, hard negatives, uncited positives, and per-query temporal corpora.

The benchmark is especially significant because it evaluates discovery across intellectual boundaries. Nearly half of credited papers come from outside the source project’s subject area, and many SubQ positives are uncited and share no references with the source paper. These properties make the task closer to exploratory research support than to citation recommendation or claim verification.

Several limitations constrain interpretation. The source papers are concentrated in AI and adjacent computer-science areas, so transfer to chemistry, biology, physics, social science, humanities, and engineering is unestablished. The source papers are recent, and more than 100 received major-conference distinctions, which may bias the dataset toward prominent research projects and authors with substantial visibility.

Temporal filtering reduces leakage but does not remove all historical bias. The corpus may contain a small number of papers dated 2025 or later because they were resolved references, and recent cohorts have less opportunity to accumulate citations or establish discoverability. The corpus is also dominated by arXiv and contains a smaller number of Semantic Scholar or OpenAlex records, producing source-coverage and accessibility biases.

Author judgments are valuable but not absolute ground truth. They are retrospective assessments made by researchers who know the completed project. The benchmark attempts to reconstruct early-stage questions and withhold final solutions, but author judgments may still reflect hindsight, memory, selective attribution, or unrecorded intellectual interactions. Positive and negative rationales mitigate this problem by documenting why papers were considered useful or non-catalytic, but the judgments remain situated and project-specific.

The benchmark’s metrics also impose a particular view of usefulness. Recall@20 rewards recovering an author’s labeled papers but does not measure whether a system’s explanations are scientifically persuasive, whether the retrieved work changes a user’s research behavior, or whether the system helps produce better research outcomes. nDCG similarly evaluates ranking position, not the quality of the intellectual connection.

Future research directions include training retrievers on author-provided rationales, constructing cross-disciplinary and non-computer-science versions, evaluating explanation faithfulness, studying query reformulation, calibrating uncertainty, and measuring downstream effects on research planning. Useful systems could preserve multiple evidence layers: semantic similarity, methodological transfer, limitation alignment, cross-domain analogy, empirical support, and author rationale.

A mature ScholarCatalyst system would likely combine several representations rather than collapse them into a single relevance score:

Dq+={papers judged by the author to offer useful inspiration for q},\mathcal{D}_q^+=\{\text{papers judged by the author to offer useful inspiration for }q\},0

This representation would allow users to ask not only which papers are relevant, but why they may matter: whether they provide an adaptable method, expose a limitation, offer evidence, or enable a conceptual bridge.

ScholarCatalyst therefore defines a demanding benchmark for scientific information retrieval. Its central claim is not that citation networks, embeddings, or agents are sufficient individually, but that research-support systems must learn a deeper relation between an unfinished problem and prior ideas that could change its trajectory.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to ScholarCatalyst.