---
title: ScholarCatalyst Benchmark
url: https://www.emergentmind.com/topics/scholarcatalyst
type: topic
---

# ScholarCatalyst Benchmark

ScholarCatalyst is a benchmark for retrieving papers whose ideas could advance an early-stage research project, rather than merely papers with topical or semantic similarity. It uses author-provided judgments from 184 researchers associated with 207 recent computer-science papers, constructing 894 research queries and 4,845 validated positive query–paper pairs. The benchmark evaluates whether retrieval systems can identify methods, limitations, phenomena, conceptual framings, evidence, and other forms of intellectual contribution that researchers considered capable of moving a project forward [2602.18479].

## 1. Scope and conceptual foundations

ScholarCatalyst distinguishes **topical relevance** from **catalytic relevance**. Topical relevance concerns shared terminology, entities, tasks, methods, or subject areas. Catalytic relevance concerns whether a paper offers an idea that could help formulate, redirect, investigate, or solve a research problem. A catalytic paper may provide an adaptable method, expose a limitation, supply empirical evidence, suggest a conceptual reframing, or connect apparently separate fields.

The benchmark is motivated by the observation that conventional literature-search systems and citation graphs incompletely represent the intellectual origins of research. A paper may influence a project without being cited, while a cited paper may be topically related but not intellectually important to the project. ScholarCatalyst therefore treats inspiration as a latent relation between an initial research question and a prior paper, inferred from expert author judgments rather than citation status alone.

Each benchmark query describes the state of knowledge at the beginning of a project, the unresolved problem, and the open question confronting the researchers, while withholding the eventual solution. The system must rank papers that were available by the time the project began. The source paper, completed solution, and source bibliography are withheld in the principal query-only setting.

The benchmark operationalizes a component of scientific “research taste”: identifying which prior ideas matter for a particular research trajectory, including connections across subfields. This does not imply that catalytic relevance is an intrinsic, universal property of a paper. The same paper may serve as a method source in one project, evidence in another, a limitation motivating a third, or a hard negative in a fourth.

## 2. Dataset construction and temporal design

The dataset contains 184 researchers, 207 source papers published during 2025–2026, and 894 queries. The source projects are concentrated in artificial intelligence and adjacent computer-science fields.

| Source-paper domain | Papers | Percentage |
|---|---:|---:|
| Machine learning (cs.LG) | 66 | 31.9% |
| Computer vision (cs.CV) | 46 | 22.2% |
| Computation and language (cs.CL) | 38 | 18.4% |
| Artificial intelligence (cs.AI) | 23 | 11.1% |
| Robotics (cs.RO) | 20 | 9.7% |
| Other | 14 | 6.8% |

More than 100 of the 207 source papers had received an oral, spotlight, or award distinction at a major AI conference. Recruitment targeted first, co-first, and corresponding authors through conference-paper lists, direct outreach, and an open call.

Each source project contributes a **CoreQ** and one or more **SubQs**. A CoreQ is a broad question concerning which prior ideas could help address the project’s central problem. A SubQ focuses on a distinct research direction or subfield that informed the project.

| Query type | Queries | Average positives | Cited / uncited positives | Average hard negatives | Average length |
|---|---:|---:|---:|---:|---:|
| CoreQ | 207 | 3.19 | 95.5% / 4.5% | 10.01 | 131.8 tokens |
| SubQ | 687 | 6.09 | 56.4% / 43.6% | 6.88 | 63.5 tokens |

The retrieval corpus contains approximately 191,000 papers, with a detailed count of 190,896. It includes resolved references from source papers and approximately 181,000 additional arXiv papers from 2020–2024, principally from cs.CV, cs.LG, cs.CL, cs.RO, and cs.AI. Of the corpus, 189,296 papers, or 99.2%, come directly from arXiv; 1,600 additional records come from Semantic Scholar or OpenAlex when cited works were unavailable on arXiv.

For a source paper $S$, the admissible search set is defined by publication date:

$$
\mathcal{P}_{<S}=\{p\in\mathcal{P}:\operatorname{date}(p)\leq\operatorname{date}(S)\}.
$$

This temporal constraint prevents systems from exploiting later descriptions of the source paper or its solution. Web search is disabled in the principal setting, and the principal agent backbones have knowledge cutoffs before the source papers. The median searchable corpus contains approximately 190,574 papers per query.

The corpus is distributed as follows:

| Corpus category | Papers | Percentage |
|---|---:|---:|
| cs.CV | 64,919 | 34.0% |
| cs.LG | 57,151 | 29.9% |
| cs.CL | 33,255 | 17.4% |
| cs.RO | 18,922 | 9.9% |
| cs.AI | 10,933 | 5.7% |
| Other arXiv categories | 4,116 | 2.2% |
| Semantic Scholar/OpenAlex | 1,600 | 0.8% |

The benchmark contains 4,845 author-validated positive query–paper pairs. CoreQs contain fewer positives and are longer and broader than SubQs. SubQs contain a substantially larger fraction of uncited positives, making them particularly important for evaluating discovery beyond bibliographic expansion.

## 3. Annotation, candidate generation, and inspiration types

ScholarCatalyst uses an automated pre-annotation pipeline followed by author validation. Starting from an arXiv identifier, the pipeline obtains metadata and source files through the arXiv API, parses the source paper, resolves bibliography entries to canonical records, and reconstructs CoreQs, SubQs, candidate inspirations, and rationales using Gemini 3.1 Pro.

Additional potentially uncited candidates are retrieved with BM25 and Qwen3-Embedding-8B. The candidate pool is reranked with Gemini 3.6 Flash, and approximately 10 candidates per query are retained for author review.

The pre-annotation pipeline is useful but insufficient. Reconstructed CoreQs were rated accurate or mostly accurate in 98.1% of cases, and SubQs in 95.9%. Approximately 85.1% of originally cited SubQ candidates were retained as positives, while 35.1% of pooled non-cited candidates were judged genuine inspirations. For automatically nominated CoreQ key inspirations, precision was 58.2%, recall was 26.0%, and the complete nominated set exactly matched the author’s set in only 9.2% of cases.

Authors can validate, rewrite, or replace the generated questions and candidate lists. For each candidate paper, they indicate whether it offers an insightful idea for pursuing the query and provide a rationale. Positive and negative sets can be represented as:

$$
\mathcal{D}_q^+=\{\text{papers judged by the author to offer useful inspiration for }q\},
$$

$$
\mathcal{D}_q^-=\{\text{reviewed papers that were topically related but not inspirational}\}.
$$

A manual consistency review found that 764 of 894 queries, or 85.5%, passed unchanged. The remaining queries were lightly reworded to remove wording that revealed the eventual solution or represented the research as completed.

Hard negatives are topically related papers that were either cited by the source paper or ranked highly by a retrieval system. They are not randomly selected irrelevant documents. This construction requires systems to distinguish intellectually useful work from plausible but non-catalytic neighboring research.

The benchmark identifies nine multi-label inspiration types:

1. **Method**: a method, component, formulation, or principle is adopted or adapted.
2. **Generalization**: an idea is extended to a new task, modality, domain, or setting.
3. **Limitation**: a limitation or failure motivates the new work.
4. **Framework**: an overall framework or system is reused.
5. **Phenomenon**: an observed empirical effect motivates investigation.
6. **Reframing**: a new conceptual perspective is supplied.
7. **Motivation**: the paper motivates the problem or direction.
8. **Evidence**: empirical evidence supports a research direction.
9. **The displayed taxonomy is described as containing nine types**, with motivation and evidence completing the list in the supplied description.

Rationales receive 1.69 labels on average. More than one type applies in 55.8% of cases, and at least three labels apply in 12.7% of cases. The most frequent roles include adapted methods, supporting evidence, generalization, and limitations that motivate new approaches.

Approximately 43.6% of SubQ positives were not cited by the source paper. Of these uncited positives, 67.8% shared no references with the source paper. Among cited positives, only 46.2% of CoreQ positives and 34.0% of SubQ positives received a “uses” or “extends” citation-intent label. In addition, 49.5% of credited papers came from outside the source project’s subject area.

These results demonstrate that citation status is an incomplete proxy for scientific inspiration. The benchmark’s positive labels instead encode author-level judgments about transferable intellectual utility.

## 4. Retrieval task, metrics, and baselines

For each query $q$ associated with source paper $S$, a system must rank the temporally admissible corpus:

$$
\operatorname{rank}(q,\mathcal{P}_{<S}),
$$

so that the author-validated positive set appears near the top.

Binary relevance is defined as:

$$
\operatorname{rel}(q,p)=
\begin{cases}
1, & p\in\mathcal{D}_q^+,\\
0, & p\notin\mathcal{D}_q^+.
\end{cases}
$$

For a ranked list $L_q=(p_1,\ldots,p_k)$, Recall@$k$ is:

$$
\operatorname{Recall@}k(q)=
\frac{
\left|\mathcal{D}_q^+\cap\{p_1,\ldots,p_k\}\right|
}{
|\mathcal{D}_q^+|
}.
$$

Aggregate Recall@$k$ is the average across queries. Thus, Recall@20 measures the fraction of an author’s positive papers appearing among the first 20 retrieved papers, rather than the fraction of queries for which at least one positive is retrieved.

The benchmark also reports normalized discounted cumulative gain:

$$
\operatorname{DCG@}k(q)=
\sum_{i=1}^{k}
\frac{2^{\operatorname{rel}(q,p_i)}-1}
{\log_2(i+1)},
$$

and:

$$
\operatorname{nDCG@}k(q)=
\frac{\operatorname{DCG@}k(q)}
{\operatorname{IDCG@}k(q)}.
$$

Reported metrics include nDCG@20 and Recall@5, Recall@20, and Recall@100.

The evaluated retrieval baselines include BM25, LateOn 0.1B, SPECTER2, SciNCL, OpenScholar, Qwen3-Embedding-4B, Qwen3-Embedding-8B, Gemini Embedding 2, GTE-Qwen2-7B-Instruct, ReasonEmbed, Inf-Retriever-v1-Pro, NV-Embed-Reasoning-3B, and DIVER-Retriever-4B.

Embedding-based ranking can be represented conceptually as:

$$
s(q,p)=\cos(f_q(q),f_p(p)),
$$

where $f_q$ and $f_p$ produce query and paper representations. Papers are ranked by decreasing similarity. BM25 performs lexical matching, whereas LateOn aggregates token-level similarities.

The Qwen3 retrieval instruction is:

> Given a scientific research question, retrieve papers that could inspire or contribute to answering the question, including relevant background knowledge, methods, and ideas.

The inclusion of this instruction emphasizes that the target relation is not ordinary topical similarity. It explicitly requests papers that could contribute ideas, methods, background, or conceptual resources.

For agentic tool-calling search, each candidate is scored from 0 to 10 according to how much its ideas could help pursue the query. If $C_q$ is the candidate pool and $g_\theta(q,p)$ is the model’s usefulness score, the approximate ranking is:

$$
\operatorname{rank}(q)=
\operatorname{sort}_{p\in C_q}g_\theta(q,p).
$$

Tournament reranking divides candidates into batches of 20 and advances the top four from each batch.

## 5. Agentic search and reported findings

The benchmark compares embedding retrieval with agentic systems that use retrieval as a tool. The principal reported result is that agentic search does not outperform embedding retrieval: the agent achieves Recall@20 of 0.42, compared with 0.48 for embedding retrieval. An agent built on Claude Fable 5.1 reaches only 0.51 Recall@20, despite the possibility that the model may have encountered completed source papers during pretraining.

This finding suggests that tool use and iterative search do not automatically produce expert-level retrieval. An agent may call the same retriever repeatedly without acquiring the latent judgment required to distinguish catalytic relevance from topical relevance. A plausible implication is that agentic systems require specialized training on expert rationales, hard-negative discrimination, query reformulation, and cross-domain analogy rather than merely access to stronger general-purpose language models.

The reported result also separates **search orchestration** from **relevance judgment**. An agent can plan searches, invoke tools, read candidate papers, and rerank results, yet still fail to identify the papers that an expert regards as intellectually catalytic. The missing capability is not necessarily retrieval coverage alone; it may involve recognizing why a paper’s idea transfers to a partially specified research problem.

The benchmark’s evaluation should not be interpreted as showing that agents are universally inferior to embedding systems. The comparison is specific to the ScholarCatalyst task, corpus, temporal restrictions, models, and reported configurations. It does show that agentic search provides no automatic advantage under these conditions.

## 6. Scientific significance, limitations, and future directions

ScholarCatalyst contributes an author-validated evaluation framework for a form of literature retrieval that is difficult to capture with citations, keywords, or ordinary semantic similarity. Its principal innovations are the use of early-stage research questions, author-supplied rationales, hard negatives, uncited positives, and per-query temporal corpora.

The benchmark is especially significant because it evaluates discovery across intellectual boundaries. Nearly half of credited papers come from outside the source project’s subject area, and many SubQ positives are uncited and share no references with the source paper. These properties make the task closer to exploratory research support than to citation recommendation or claim verification.

Several limitations constrain interpretation. The source papers are concentrated in AI and adjacent computer-science areas, so transfer to chemistry, biology, physics, social science, humanities, and engineering is unestablished. The source papers are recent, and more than 100 received major-conference distinctions, which may bias the dataset toward prominent research projects and authors with substantial visibility.

Temporal filtering reduces leakage but does not remove all historical bias. The corpus may contain a small number of papers dated 2025 or later because they were resolved references, and recent cohorts have less opportunity to accumulate citations or establish discoverability. The corpus is also dominated by arXiv and contains a smaller number of Semantic Scholar or OpenAlex records, producing source-coverage and accessibility biases.

Author judgments are valuable but not absolute ground truth. They are retrospective assessments made by researchers who know the completed project. The benchmark attempts to reconstruct early-stage questions and withhold final solutions, but author judgments may still reflect hindsight, memory, selective attribution, or unrecorded intellectual interactions. Positive and negative rationales mitigate this problem by documenting why papers were considered useful or non-catalytic, but the judgments remain situated and project-specific.

The benchmark’s metrics also impose a particular view of usefulness. Recall@20 rewards recovering an author’s labeled papers but does not measure whether a system’s explanations are scientifically persuasive, whether the retrieved work changes a user’s research behavior, or whether the system helps produce better research outcomes. nDCG similarly evaluates ranking position, not the quality of the intellectual connection.

Future research directions include training retrievers on author-provided rationales, constructing cross-disciplinary and non-computer-science versions, evaluating explanation faithfulness, studying query reformulation, calibrating uncertainty, and measuring downstream effects on research planning. Useful systems could preserve multiple evidence layers: semantic similarity, methodological transfer, limitation alignment, cross-domain analogy, empirical support, and author rationale.

A mature ScholarCatalyst system would likely combine several representations rather than collapse them into a single relevance score:

$$
\operatorname{CatalyticProfile}(q,p)=
(\operatorname{Method},\operatorname{Generalization},\operatorname{Limitation},
\operatorname{Framework},\operatorname{Phenomenon},
\operatorname{Reframing},\operatorname{Motivation},\operatorname{Evidence}).
$$

This representation would allow users to ask not only which papers are relevant, but why they may matter: whether they provide an adaptable method, expose a limitation, offer evidence, or enable a conceptual bridge.

ScholarCatalyst therefore defines a demanding benchmark for scientific information retrieval. Its central claim is not that citation networks, embeddings, or agents are sufficient individually, but that research-support systems must learn a deeper relation between an unfinished problem and prior ideas that could change its trajectory.

Source: https://www.emergentmind.com/topics/scholarcatalyst