Papers
Topics
Authors
Recent
Search
2000 character limit reached

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Published 1 Oct 2026 in cs.AI, cs.CL, and cs.IR | (2610.02202v1)

Abstract: What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Summary

Problem formulation and contribution

“ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research” (2610.02202) introduces a benchmark for a retrieval problem that is distinct from conventional scientific-document search. The objective is not to retrieve papers that support a claim, match a query facet, or bear a known citation relation. It is to identify prior work whose ideas could materially advance an early-stage research project, including papers that the project authors did not cite and papers whose topical or lexical relationship to the query is weak.

The paper frames this capability as a component of research judgment or “research taste.” A completed publication exposes the final problem formulation, method, experiments, and bibliography, but it only partially records the intellectual process through which the project developed. In particular, bibliographies do not reliably distinguish papers that supplied decisive conceptual or methodological insight from papers cited for context, comparison, or convention. The benchmark therefore treats researchers’ retrospective accounts as a source of relevance supervision.

The central empirical claim is that current retrieval systems perform poorly on this task, and that adding LLM-based search behavior does not resolve the principal bottleneck. In the main temporally controlled setting, the strongest embedding retrievers recover at most approximately half of the author-identified catalyst papers in their top 20 results. Agentic systems that repeatedly query the same similarity-based retriever perform no better, and often slightly worse, than the underlying retriever. The results support a specific diagnosis: systems frequently fail to place useful inspiration papers into the candidate pool, rather than merely failing to recognize their relevance after retrieval.

Benchmark construction

ScholarCatalyst contains 207 source papers contributed by 184 researchers and yields 894 author-validated queries. The source papers were published in 2025–2026 and are concentrated in computer science and AI, with the largest groups in machine learning, computer vision, computational linguistics, and artificial intelligence. Each project contributes one CoreQ and one or more SubQs.

A CoreQ represents the project’s central research question before its solution was known. A SubQ represents a more localized research direction or subfield through which the broader project could be advanced. This distinction is methodologically important: a paper may be useful for the project’s overarching conceptual organization without being the best paper for a particular technical direction, and vice versa.

Each query is paired with author-labeled positives and hard negatives. A positive is a prior paper that the authors judge to offer an insight that could inspire or help advance the query. A hard negative is topically related but was judged not to provide such an insight. The benchmark consequently evaluates discrimination between scientific usefulness and topical relatedness. The average CoreQ has 3.19 positives and 10.01 hard negatives; the average SubQ has 6.09 positives and 6.88 hard negatives. SubQ positives are substantially more likely to be uncited: 43.6% are absent from the source paper’s bibliography.

The retrieval corpus contains approximately 191,000 computer-science papers, predominantly arXiv articles from 2020–2024, supplemented by resolved references from the source papers. For each query, documents published after the corresponding source paper are excluded. The source paper, its bibliography, and external web access are withheld in the principal evaluation. This design approximates the information state available during project development and prevents direct exploitation of the completed paper.

The benchmark is assembled through an automated pipeline that parses source papers, resolves citations, proposes research questions and candidate inspirations, and presents these materials to authors for validation and revision. The pipeline’s reconstructed questions were rated accurate or mostly accurate for 98.1% of CoreQs and 95.9% of SubQs. Authors retained 85.1% of initially proposed cited candidates as positives, while 35.1% of pooled non-cited candidates were promoted to positives. These figures show both that citations are a productive source of candidate papers and that they provide incomplete supervision for inspiration.

Inspiration is not equivalent to similarity or citation

The benchmark’s most important conceptual result is that catalyst relations are heterogeneous and poorly represented by standard similarity functions. The authors classify 663 key-inspiration rationales into a nine-type taxonomy. The most common relations involve adapting a method, using empirical evidence, generalizing an idea to a new setting, and responding to a limitation. These relations are often concurrent: 55.8% of rationales receive at least two labels, and 12.7% receive at least three. A single source paper can therefore influence a project through several mechanisms that are not reducible to shared terminology.

The similarity analysis directly supports this interpretation. Hard negatives are at least as similar to queries as positives under both lexical and embedding-based measures. For CoreQ, mean lexical similarity is 28.7 for positives and 29.2 for hard negatives, while mean semantic similarity is 64.5 and 66.2, respectively. For SubQ, lexical similarity is effectively identical at 34.5 and 34.4, and semantic similarity slightly favors positives at 65.1 versus 64.2. Thus, a paper can be scientifically useful despite appearing semantically distant, while a highly similar paper can fail to provide an actionable idea.

Citation status is similarly inadequate as a proxy. Although most CoreQ positives are cited, only 46.2% of cited CoreQ positives carry a “uses” or “extends” citation-intent label. The corresponding figure for SubQ positives is only 34.0%. Moreover, 43.6% of SubQ positives are uncited, and 67.8% of those uncited positives share no references with the source paper. The implication is direct: systems trained only on citation graphs, citation intents, or document similarity are optimized against signals that systematically omit the target relation.

The context dependence of inspiration further complicates benchmark design. The same prior paper can function as a baseline in one project, a cross-domain generalization source in another, and a structural premise in a third. Relevance is therefore not an intrinsic property of a paper-query pair independent of the project context. It depends on what the researcher is attempting to accomplish and on which scientific transformation the prior contribution enables.

Retrieval baselines and principal results

The evaluation covers BM25, late-interaction retrieval, scientific-document embeddings, general-purpose embeddings, reasoning-oriented embeddings, and three agentic search configurations. The main agentic systems include a grep-based agent, a tool-calling search-and-select agent, and a deep-research agent with subquestion decomposition and iterative search. GPT-4.1 and o3 satisfy the pre-cutoff requirement in the main comparison.

General-purpose embedding models are consistently strongest. Qwen3-Embedding-8B achieves Recall@20 of 0.37 on CoreQ and 0.51 on SubQ, while Gemini Embedding 2 achieves 0.38 and 0.46. The corresponding Recall@100 values for Gemini Embedding 2 are 0.62 and 0.70. These results reveal a substantial difference between query types: SubQs are easier, plausibly because they are shorter, more technically localized, and associated with larger positive sets.

System CoreQ R@20 SubQ R@20
BM25 0.23 0.26
SPECTER2 0.20 0.23
OpenScholar 0.23 0.24
Qwen3-Embedding-8B 0.37 0.51
Gemini Embedding 2 0.38 0.46
GPT-4.1 tool-calling agent 0.37 0.43
o3 deep-research agent 0.33 0.38

Scientific-document retrievers perform particularly poorly relative to general-purpose embeddings. OpenScholar trails the strongest general-purpose systems by at least 16 percentage points on CoreQ and 27 points on SubQ. Reasoning-optimized embedding models also fail to improve over general-purpose embedding retrieval. This is a consequential negative result: objectives designed for reasoning-intensive retrieval do not necessarily encode the latent relation between a research question and a prior paper that offers a transferable or motivating idea.

The agentic systems do not improve over their retrieval substrate. The GPT-4.1 tool-calling agent reaches CoreQ and SubQ Recall@20 of 0.37 and 0.43, compared with 0.38 and 0.46 for Gemini Embedding 2. The o3 deep-research agent reaches 0.33 and 0.38. Grep performs substantially worse, with Recall@20 of only 0.06 on CoreQ and 0.09 on SubQ. Its low performance reflects both weak lexical matching and inadequate candidate coverage.

The paper’s comparison with Claude Fable 5.1 is explicitly an upper reference rather than a valid leakage-controlled baseline. Its June 2026 knowledge cutoff may include the source papers or secondary descriptions of them. Even under this advantage, the model reaches only 0.51 CoreQ and 0.48 SubQ Recall@20 with tool-calling search. The result is consistent with the claim that exposure to completed papers does not automatically produce reliable catalyst retrieval, although it cannot isolate memorization from general model capability.

Candidate coverage as the dominant bottleneck

The paper separates candidate generation from candidate recognition through trajectory analysis and oracle candidate injection. Gemini Embedding 2 places 70% of all positives in its top 100, but agents recover very few positives outside that set. For the GPT-4.1 tool-calling agent, 40% of all positives are both in the retriever’s top 100 and in the agent’s top 20, while 30% are in the retriever’s top 100 but missed by the agent. Only 1% lie outside the retriever’s top 100 and are recovered by the agent. The corresponding outside-pool recovery for the deep-research agent is 2%.

These figures establish that iterative reasoning rarely compensates for retrieval failure. Approximately half of the positives are not encountered by an agent during its search trajectory. The first query, usually the original research question, accounts for roughly two-thirds of the gold papers that agents do encounter. Follow-up searches add relatively few new positives and can cause query drift toward narrower or more generic subtopics.

The reranking experiments reinforce the same diagnosis. When missing positives are injected into candidate pools, stronger LLM rerankers benefit substantially more than weaker rerankers. For example, Claude Fable’s standard Recall@5 on CoreQ rises from 0.31 with 20 candidates to 0.60 under oracle injection, whereas its standard performance is constrained by what the retriever initially surfaces. The implication is not that reranking is unimportant. Rather, recognition capacity becomes useful only after candidate generation has recovered papers whose relevance is latent under ordinary similarity.

The agent analyses also show that recognition is not entirely absent. Among positive papers that the tool-calling agent ranks in its top 20, 76% of comparable model-generated rationales describe the same inspiration relationship as the author rationale. Almost all remaining cases describe a different but valid relationship. This evidence suggests that, conditional on receiving a catalyst paper, the model can often explain its usefulness. The unresolved problem is getting that paper into the model’s deliberative context.

Context, reading, and query reformulation

Additional source context improves search, but primarily by guiding retrieval rather than by enabling closed-book recall. Providing the source paper’s title and abstract increases Gemini 3.7 Flash tool-calling performance on CoreQ from 0.37 to 0.48 in one cutoff slice and from 0.41 to 0.52 in another. Web access changes Recall@20 by at most approximately one percentage point. Without corpus search, closed-book performance is only 0.31 on CoreQ and 0.24 on SubQ in the reported aggregate analysis.

Providing the source bibliography nearly reveals the answer because it contains most of the gold papers. Bibliography-assisted Recall@20 reaches 0.68 and 0.81 in the two CoreQ cutoff slices, while the bibliography alone reaches 0.60. This condition is intentionally an upper reference, not a deployable solution: it violates the benchmark’s pre-discovery setting and demonstrates why the source paper and its citations must be withheld.

Reading more of retrieved papers does not solve the problem. Adding introductions, methods, bibliographies, or full papers changes Recall@20 by at most a few points, and the best SubQ policy remains below retriever-only performance in the relevant analysis. The result localizes the bottleneck before document comprehension: full-text reasoning cannot help when the system does not know which documents to inspect.

LLM-based query rewriting is also ineffective. Single-query expansion improves CoreQ Recall@20 by four points but reduces SubQ recall by five points. Multi-query generation slightly reduces recall for both query types, while HyDE reduces CoreQ recall from 0.40 to 0.23 and SubQ recall from 0.42 to 0.22 in the reported experiment. These failures are consistent with the similarity analysis: the problem is not merely that the query uses the wrong wording. The embedding space itself often fails to place the catalyst paper near the research question.

Limitations and open questions

The benchmark is restricted to computer science, with substantial concentration in AI and machine learning, and its source papers are drawn primarily from 2025–2026 conference research. Transfer to other disciplines, especially fields with different publication practices and citation conventions, is unresolved. The corpus is also approximately 191,000 papers, much smaller than the literature accessible to researchers in practice. Because the benchmark is therefore an easier version of full-literature retrieval, the reported scores may overestimate real-world difficulty.

The labels are retrospective judgments by project authors. Hindsight can alter recollections of which prior work mattered, and author judgments may conflate historical influence with counterfactual usefulness. The paper acknowledges that the benchmark targets author-credited inspiration rather than an objectively identifiable causal relation. A preliminary three-author comparison found that coauthors recovered 43–60% of a first author’s positives at 84–88% precision, suggesting meaningful but imperfect inter-annotator agreement. This comparison is limited to one project and cannot establish a general human ceiling.

The automated construction pipeline also introduces selection effects. Candidate discovery begins with cited papers and sparse or dense retrieval, after which authors review a limited candidate set. Although the pipeline recovers many non-cited positives, papers absent from this candidate-generation process can remain false negatives. The benchmark’s positive sets should therefore be interpreted as author-validated positives, not exhaustive sets of all papers that could have helped.

Several questions remain open within the paper’s specific framework. It is not yet known whether a retriever trained directly on author rationales and counterfactual inspiration relations can improve candidate coverage, whether citation-graph expansion can recover cross-domain catalysts without reintroducing topical bias, or whether direct corpus interaction can outperform similarity-based tools when the corpus is sufficiently large and structured. The appropriate performance ceiling is also unsettled: expert agreement is clearly above current model performance, but the task’s intrinsic ambiguity prevents a straightforward analogue of standard classification accuracy.

Conclusion

ScholarCatalyst formalizes a neglected retrieval objective: finding prior papers that could catalyze progress on an early-stage research question. Its author-validated annotations show that inspiration is frequently uncited, cross-disciplinary, context-dependent, and weakly correlated with lexical or semantic similarity. On the resulting benchmark, general-purpose embedding retrieval outperforms scientific-specific and reasoning-oriented retrievers, while current search agents add little because they usually inherit the same candidate-coverage failures. The paper therefore identifies expert-level candidate generation, rather than merely deeper reading or longer agentic reasoning, as the central unresolved problem in retrieval for scientific inspiration.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Explain it Like I'm 14

1. What is this paper about?

This paper introduces ScholarCatalyst, a test for AI systems that search scientific papers.

The main question is:

Can an AI find the older research papers that might give scientists the key idea they need to solve a new problem?

Scientists do not only search for papers that use the same words as their question. Sometimes a very useful paper comes from a different area and contains an idea that can be adapted. The authors call these helpful papers “catalyst papers” because they can spark or speed up new research.

For example, imagine someone wants to build a better robot. The most useful paper might not be about robots at all. It might describe how ants organize themselves, giving the researcher a new idea.

2. What did the researchers want to find out?

The researchers focused on several questions:

  • Can current AI search systems find papers that scientists say were useful or could have been useful?
  • Are ordinary methods, such as matching similar words or meanings, good enough?
  • Do more advanced AI “search agents” perform better than simpler search systems?
  • Are the most inspiring papers usually the ones that look most similar to the research question?
  • What kinds of ideas do catalyst papers provide?

The researchers were especially interested in early-stage research questions. These are questions written as they might have appeared before the researchers knew the final answer. This prevents the search system from simply looking for words from the finished paper.

3. How was the study carried out?

Building the benchmark

The team asked 184 researchers about 207 recent computer science and artificial intelligence projects.

For each project, the researchers:

  1. Described the project’s original research question.
  2. Wrote a broader question about the project’s main problem, called a CoreQ.
  3. Wrote more focused questions about particular research areas, called SubQs.
  4. Looked at possible earlier papers.
  5. Marked which papers had ideas that could help advance the project.
  6. Explained why each selected paper was useful.

The final benchmark contained:

  • 894 research questions
  • About 191,000 earlier papers
  • Nearly 900 author-identified questions
  • Helpful papers, called positive papers
  • Related but unhelpful papers, called hard negatives

A hard negative is like a wrong answer that looks very convincing. For example, a paper might discuss almost exactly the same topic but offer no useful idea for the project.

Comparing different search systems

The researchers tested several kinds of systems:

  • BM25, which mainly looks for matching words
  • Embedding retrievers, which compare the meaning of the question and papers
  • Scientific-paper search systems
  • AI search agents that search repeatedly, break questions into smaller parts, and read results
  • An advanced LLM called Claude Fable 5.1, used only as an extra comparison

The papers available to each system were limited to those published before the research project was completed. This made the test more realistic: the system could not simply use later papers that described the final discovery.

Measuring success

The researchers used Recall@20. This measures how many of the correct papers appeared in the system’s first 20 results.

For example, if the authors identified 10 useful papers and the search system found 5 of them in its top 20 results, its recall would be 50%.

4. What were the main findings?

Current systems miss many important papers

The strongest ordinary search systems found only about 48% of the author-identified catalyst papers in their top 20 results.

This means that more than half of the papers that experts considered useful were usually missing from the first 20 results.

Search was harder for the broad, central questions than for the more specific subfield questions.

More complicated AI agents did not solve the problem

The researchers expected AI search agents to do better because they can:

  • Rewrite questions
  • Search several times
  • Break a question into smaller parts
  • Read and compare papers
  • Think about the results

However, these agents did not clearly beat simpler embedding search. One result found about 42% of the useful papers, while the simpler search found about 48%.

This happened because the agents still depended on a search tool to produce possible papers. If the search tool failed to show an important paper, the AI agent usually could not consider it.

This is similar to asking a student to choose the best book from a library shelf when the librarian has already placed the most useful books somewhere else. Even a very clever student cannot choose a book that was never put on the shelf.

Similarity is a weak clue

The useful papers were not always the papers that looked most similar to the question.

The authors found that rejected papers could be just as similar to the research question as the helpful papers. This means that matching words or general topics is not enough.

A catalyst paper might help because it:

  • Introduces a method that can be adapted
  • Provides evidence supporting a research direction
  • Applies an idea in a new setting
  • Shows a limitation that inspires a new approach

These useful connections can be difficult for a computer to notice because they may not be directly stated in the question.

Citations do not tell the whole story

The papers cited in a finished research article are not a complete record of what inspired the work.

For example:

  • About 43.6% of useful papers for subfield questions were not cited in the final paper.
  • Some useful papers came from outside the project’s main subject area.
  • A paper can be important because it inspired an idea, even if the researchers did not directly use its method or cite it.

This shows why looking only at citation lists is not enough to understand how research develops.

Better access to the right papers matters more than reading more deeply

The study suggests that the biggest problem is not that AI cannot understand papers after finding them. The bigger problem is that AI often fails to find the right papers in the first place.

When the researchers gave an AI the project’s bibliography, its performance improved greatly because the bibliography already contained many of the useful papers.

However, allowing the AI to read longer sections of retrieved papers usually helped only a little. This supports the idea that finding the right candidates is the main challenge.

Even a newer model missed many useful papers

Claude Fable 5.1 was tested as a special comparison because it may have seen some of the papers during training. Even so, it found only about half of the useful papers in its top 20 results.

This suggests that simply making LLMs larger or newer may not be enough. They may need special training to recognize which ideas could help a new research project.

5. Why are these results important?

The paper shows that scientific research involves more than finding papers about the same topic.

A strong scientist must often make a creative connection:

“This paper is not directly about my problem, but its idea could help me solve it.”

Current AI systems are much better at finding papers with similar words or meanings than at making these deeper connections.

ScholarCatalyst is important because it provides a way to measure this ability. Instead of asking whether a paper is merely related to a topic, it asks whether experts believe the paper could actually help move a research project forward.

6. What could this mean for the future?

In the future, better scientific AI systems might help researchers by:

  • Finding useful ideas from many different fields
  • Suggesting papers that researchers would not normally discover
  • Helping turn a vague question into a clearer research plan
  • Revealing connections between areas of science that are usually separate
  • Saving researchers time when searching through millions of papers

However, these systems need a kind of research intuition or “research taste.” They must learn to recognize valuable ideas even when the paper does not look obviously similar to the question.

The authors also point out that this could be especially useful because nearly half of the useful papers identified by researchers came from outside their main subject area. An AI system that can search across many fields could therefore discover ideas that no single human researcher has had time to read.

Simple conclusion

ScholarCatalyst shows that finding truly inspiring research papers is much harder than finding papers on the same topic. Current AI search systems can retrieve some useful papers, but they miss many others, and complicated search agents do not yet solve the problem.

The research provides a useful test for building better scientific AI. If future systems learn to recognize important connections between distant ideas, they could help scientists discover breakthroughs hidden among the enormous number of papers published every year.

Practical Applications

Immediate Applications

The paper’s findings support several applications that can be deployed with current retrieval infrastructure, provided they are treated as decision-support tools rather than autonomous scientific authorities.

  • Research-literature discovery assistants for universities and R&D teams — Academia, software, industrial research
    • Integrate the benchmark’s workflow into scholarly search systems: collect an early-stage research question, retrieve candidate papers with dense and lexical search, and present them with explanations, rationales, and confidence indicators.
    • A practical workflow would combine BM25 and embedding retrieval, followed by human review of the top candidates. Because current systems achieve only approximately 39–51% Recall@20 depending on query type, results should be framed as possible catalysts, not comprehensive recommendations.
    • Dependencies: access to a sufficiently broad, temporally filtered corpus; high-quality metadata and abstracts; transparent ranking; and researcher verification.
  • Early-stage project-scoping and grant-writing support — Academia, public-sector research, corporate R&D
    • Researchers can use the CoreQ/SubQ distinction to separate:
    • a broad question about the central research problem, and
    • narrower searches from specific disciplinary perspectives.
    • The resulting workflow can help identify methodological precedents, empirical evidence, limitations motivating new work, and ideas that may generalize across fields.
    • This is particularly useful when preparing literature reviews, grant proposals, technical roadmaps, or project kickoffs.
    • Dependencies: the user must provide a genuinely pre-discovery description of the problem and independently assess whether retrieved papers are actually applicable.
  • Cross-disciplinary “adjacent field” search — Healthcare, robotics, energy, materials, finance, climate science
    • Since nearly half of the credited catalyst papers came from outside the project’s immediate subject area, organizations can deliberately search neighboring disciplines rather than relying only on keyword-matched literature.
    • For example, a robotics group might search computer vision, control theory, human-computer interaction, and neuroscience separately; an energy team might search optimization, materials science, economics, and power systems.
    • Tools could generate several SubQs for different technical perspectives and merge the resulting candidate lists for expert review.
    • Dependencies: broad indexing across disciplines, compatible terminology, and domain experts who can judge whether an analogy is technically meaningful rather than merely superficial.
  • Human-in-the-loop literature triage — Libraries, research offices, information services
    • The paper’s automated data-collection pipeline can be adapted into an annotation interface in which researchers:
    • 1. describe an early-stage problem,
    • 2. review candidate papers,
    • 3. label useful and non-useful papers,
    • 4. explain the reason for each judgment, and
    • 5. refine the query.
    • Libraries and research-support offices could use this process to create tailored reading lists for labs, research consortia, or emerging research areas.
    • Dependencies: researcher time, incentives for annotation, privacy protection for unpublished research questions, and mechanisms for resolving disagreement between annotators.
  • Evaluation of academic-search and scientific-agent products — Software and AI-platform development
    • SCHOLARCATALYST can serve as an evaluation suite for comparing search engines, embedding models, rerankers, and research agents on a task that is harder than topical relevance.
    • Product teams can measure Recall@5, Recall@20, Recall@100, and nDCG@20, while separately tracking:
    • candidate coverage,
    • ranking quality,
    • citation versus non-citation discovery,
    • cross-domain retrieval,
    • and explanation quality.
    • This can prevent systems from appearing effective merely because they return papers with similar vocabulary.
    • Dependencies: licensing and availability of the benchmark, protection against training-data leakage, and evaluation on newly released instances.
  • Temporal and reproducible literature search — Academia, regulated R&D, policy analysis
    • The paper demonstrates a useful method for reconstructing the literature available before a project was completed. This can be used to create “as-of-date” literature reviews and audit trails.
    • Potential applications include reproducible systematic reviews, patent and technology landscaping, historical analysis of research directions, and documentation of what information was available when a scientific or engineering decision was made.
    • Dependencies: reliable publication dates, version control for preprints, deduplication of papers, and accurate bibliographic resolution.
  • Improved researcher and student education — Education and daily academic life
    • Graduate courses can use catalyst-retrieval exercises to teach students that important prior work may not be the most textually similar paper.
    • Students can compare:
    • papers returned by keyword or embedding search,
    • papers identified by an expert,
    • and their own explanations of why a paper could advance a project.
    • This supports training in literature judgment, research taste, hypothesis formation, and critical reading.
    • Dependencies: instructor moderation, access to suitable domain-specific corpora, and explicit warnings against treating model rankings as authoritative.
  • Internal knowledge discovery in companies — Software engineering, pharmaceutical research, engineering
    • Organizations can index internal technical reports, design documents, incident reports, patents, and external publications, then use the benchmark’s annotation approach to identify documents that could inform a new project.
    • For example, an engineering team could search prior failure analyses for limitations that motivate a redesign, while a pharmaceutical team could identify methods from adjacent assay or modeling domains.
    • Dependencies: secure retrieval over confidential data, access controls, document quality, and legal review for intellectual-property or regulatory constraints.
  • Policy and funding-program design — Research policy and science administration
    • Funding agencies can use author-annotated catalyst data to study how research progress is actually connected to prior work, including uncited or cross-disciplinary influences that bibliometric indicators miss.
    • This could inform broader definitions of research impact, support interdisciplinary funding programs, and guide investments in open scholarly infrastructure.
    • Dependencies: representative sampling beyond highly visible computer-science conference papers, safeguards against retrospective bias, and careful separation of discovery support from funding decisions.

Long-Term Applications

The following applications require substantial advances in candidate coverage, expert-level relevance modeling, data scale, or validation across disciplines.

  • Expert-level scientific research agents — Academia, industrial R&D, healthcare, energy, robotics
    • A mature agent could accept a half-formed research idea, decompose it into CoreQs and SubQs, search millions of papers, identify non-obvious catalyst papers, and explain the specific transferable insight in each one.
    • Rather than simply producing a bibliography, it could organize results by inspiration type: method, evidence, limitation, generalization, or conceptual reframing.
    • Such an agent could then propose experiments or project directions grounded in the retrieved literature.
    • Dependencies: retrieval systems that overcome semantic-similarity limits; direct or richer access to full corpora; expert-validated training data; temporal controls; and reliable uncertainty estimation.
  • Training models for “research taste” and inspiration-aware retrieval — AI research and software
    • SCHOLARCATALYST can support new learning objectives in which models are trained to distinguish papers that are merely related from papers whose ideas could advance a project.
    • Potential methods include:
    • contrastive learning with author-approved positives and hard negatives,
    • preference optimization over expert rationales,
    • reinforcement learning from research-search trajectories,
    • graph models combining papers, methods, citations, and concepts,
    • and multi-stage retrieval followed by reasoning-based reranking.
    • Dependencies: much larger and more diverse annotation sets, agreement studies, controls for hindsight bias, and protection against models memorizing completed source papers.
  • Large-scale cross-domain discovery systems — Climate, medicine, materials, manufacturing, public health
    • A system with broad expert-level retrieval could identify “undiscovered public knowledge”: useful combinations of methods or findings that exist in separate literatures but have not yet been connected.
    • Examples include transferring machine-learning methods into medical imaging, adapting materials-discovery techniques to energy storage, or applying control methods from robotics to grid management.
    • Dependencies: coverage of the full scholarly and technical literature, terminology normalization, domain-specific validation, intellectual-property review, and experimental confirmation of proposed transfers.
  • Research-planning copilots that connect inspiration to execution — Laboratories and engineering organizations
    • Future tools could transform retrieved catalyst papers into an actionable project plan:
    • identify the transferable method,
    • list required data or equipment,
    • flag assumptions,
    • predict likely limitations,
    • suggest validation experiments,
    • and maintain provenance links to the supporting literature.
    • This would connect literature discovery with experiment design, simulation, code generation, and laboratory workflow management.
    • Dependencies: accurate scientific reasoning, integration with laboratory and software systems, reproducibility of cited results, and human approval before experiments or operational changes.
  • Automated technology and prior-art landscaping — Industry, patents, legal and policy analysis
    • Inspiration-aware retrieval could supplement conventional citation and keyword searches in patent landscaping, competitive intelligence, standards analysis, and technology forecasting.
    • It may uncover technically influential papers that are not highly cited, not explicitly referenced, or located outside the target industry’s usual vocabulary.
    • Dependencies: legal-grade provenance, comprehensive patent and publication databases, strict distinction between scientific inspiration and legal prior art, and expert review by patent or regulatory professionals.
  • Evidence-informed healthcare and biomedical research navigation — Healthcare and life sciences
    • In the long term, biomedical researchers could use catalyst retrieval to find methods, limitations, or evidence from adjacent areas such as genomics, imaging, clinical trials, epidemiology, and computational biology.
    • The system could help formulate translational research questions and identify promising methodological adaptations.
    • It should not directly generate clinical recommendations; retrieved ideas would require laboratory, clinical, and regulatory validation.
    • Dependencies: biomedical corpus coverage, patient-data privacy, reproducibility of findings, clinical expertise, safety review, and strict controls against unsupported medical inference.
  • Policy intelligence for emerging scientific and technological risks — Government and international organizations
    • Governments could use inspiration-aware systems to map how ideas move across fields and identify emerging research directions, dual-use concerns, or technology dependencies before they become visible through citation counts.
    • Outputs could support horizon scanning, strategic investment, standards development, and research-security assessments.
    • Dependencies: balanced global coverage, multilingual retrieval, protection against political or institutional bias, transparent evidence trails, and human-led interpretation.
  • Personal research and learning companions — Daily life and lifelong education
    • A future general-purpose assistant could help a learner move from a vague question to a structured exploration of relevant ideas, including papers from fields the learner would not normally encounter.
    • For example, someone exploring battery recycling, educational technology, or personal data privacy could receive explanations of transferable concepts rather than only popular or keyword-matched sources.
    • Dependencies: accessible explanations, reliable source attribution, protection from misinformation, adaptation to the user’s background, and clear communication that inspiration does not establish correctness or practical feasibility.
  • Continually refreshed scientific benchmarks and model-audit infrastructure — Research governance and AI safety
    • Because new papers can become benchmark instances before they enter model-training corpora, the pipeline could support rolling evaluations of scientific agents under changing knowledge conditions.
    • This would help detect leakage, measure genuine retrieval progress, and evaluate whether agents improve in breadth, cross-domain transfer, and expert agreement.
    • Dependencies: sustained author participation, standardized annotation protocols, secure temporal splits, regular corpus updates, and long-term institutional maintenance.

Glossary

  • Agentic search: Information retrieval performed by an autonomous language-model agent that iteratively formulates queries and uses search tools. “Search agents use LLM reasoning to iteratively formulate queries and invoke these retrievers as tools over multiple turns”
  • BM25: A probabilistic lexical-ranking algorithm that scores documents according to query-term occurrence and rarity. “We use Gemini 3.6. Flash (Google DeepMind, 2026b) reranks the pooled candidates for scientific relevance”
  • Candidate coverage: The extent to which a retrieval system places relevant documents in the set available for subsequent ranking or reasoning. “We attribute this gap to candidate coverage, since regardless of how a model reasons, an agent can interact with the corpus only through its search queries.”
  • Catalyst paper: A prior paper whose ideas could meaningfully inspire or advance a research project. “find its catalyst papers, by which we mean prior papers whose ideas did or could have meaningfully advanced the project”
  • Citation graph: A network representing papers as nodes and their citation relationships as edges. “The resulting one-hop citation graph links inline citations to the referenced papers' metadata and abstracts.”
  • Citation-intent annotation: A label describing the purpose or influence of one paper’s citation of another. “it derives positives from citation-intent annotations indicating methodological influence”
  • Closed-book evaluation: An evaluation setting in which a model cannot access an external corpus or search tools. “Closed-book (no corpus)”
  • Corpus: A collection of documents indexed for retrieval or analysis. “Our retrieval corpus P\mathcal{P} contains 190,896 papers.”
  • Dense retriever: A retrieval model that represents queries and documents as learned continuous vectors and compares them semantically. “Lexical retrievers and embedding retrievers rank corpus papers through term overlap or learned semantic similarity”
  • Embedding space: A continuous vector space in which texts are positioned according to learned representations. “the embedding space itself does not place inspiration papers near their queries”
  • Embedding retrieval: Retrieval based on semantic vector representations rather than exact word matching. “Agentic search does no better than embedding retrieval”
  • Faceted query: A query that targets a particular aspect or dimension of a document or information need. “Existing scientific retrieval benchmarks evaluate how well systems find evidence for claims, retrieve documents for specified facets”
  • Gold paper: A paper identified by the benchmark’s human annotators as relevant or inspirational. “Share of gold papers that each GPT-4.1 agent encounters during search”
  • Gold injection: An experimental procedure that inserts missing relevant documents into a retrieval candidate set. “which replaces negative candidates with positive papers missing from the set while keeping the pool size fixed”
  • Hard negative: A document that is related to a query but is judged irrelevant for the target task. “we also provide hard-negative documents, Dq−⊂P<S\mathcal{D}_q^- \subset \mathcal{P}_{< S}, which are related to qq but did not inspire the authors of SS.”
  • HyDE: Hypothetical Document Embeddings, a retrieval technique that generates a hypothetical answer or document and embeds it to search for real documents. “both HyDE variants substantially reduce recall”
  • Information retrieval: The study and practice of finding relevant information from a collection of documents. “Lexical retrievers rank corpus papers through term overlap or learned semantic similarity”
  • Latent relevance: Relevance that is not explicitly stated in a query or document and must be inferred. “we pose an open-ended problem that requires inferring the latent scientific relevance of papers”
  • Lexical retriever: A retrieval system that ranks documents using exact or statistical word overlap. “We include BM25 as a lexical baseline”
  • Multi-vector retriever: A retriever that represents a document or query with multiple vectors rather than a single embedding. “We evaluate lexical, multi-vector, dense, and agentic retrieval.”
  • nDCG@N: Normalized discounted cumulative gain measured at rank position NN, rewarding relevant documents placed near the top of a ranking. “normalized discounted cumulative gain (nDCG@N)”
  • One-hop citation graph: A citation network containing the direct references of a paper, without recursively expanding to further citation levels. “The resulting one-hop citation graph links inline citations to the referenced papers' metadata and abstracts.”
  • Oracle candidate pool: A candidate set assumed to contain the relevant documents, used to isolate ranking quality from initial retrieval quality. “We show later that reranking with an oracle candidate pool recovers part of this gap”
  • Pre-discovery query: A query describing a research problem before its eventual solution or discovery is known. “SCHOLARCATALYST combines queries that precede a project's solution (Pre-discovery query)”
  • Query expansion: The transformation of an original query into a longer or augmented query intended to improve retrieval. “We test whether an LLM can improve retrieval by reasoning over the original research question to generate single-query expansions”
  • Query rewriting: Reformulating a query to improve the retrieval of relevant documents. “Query rewriting does not consistently improve retrieval.”
  • Recall@N: The fraction of all relevant documents retrieved within the top NN results. “We evaluate retrieval performance using Recall@N, which measures the fraction of positive papers retrieved in the top NN results”
  • Reranking: Reordering an initial set of retrieved documents using a more sophisticated relevance model. “Stronger rerankers benefit more from better candidate coverage.”
  • Retrieval corpus: The document collection from which a retrieval system selects results. “By making the retrieval corpus P<S\mathcal{P}_{< S} contain only papers published before SS”
  • Retrieval cutoff: A temporal or model-knowledge boundary that limits which documents or information may be used. “Agent backbones must have a knowledge cutoff before every source paper”
  • Semantic similarity: Similarity between texts based on their meanings or learned representations rather than literal word overlap. “semantic similarity as cosine similarity under Qwen3-Embedding-8B”
  • Sparse retrieval: Retrieval using sparse representations, typically based on lexical features such as words or tokens. “Across sparse, dense, and multi-vector retrievers and LLM search agents”
  • Subfield-specific query (SubQ): A query that frames a central research question from the perspective of a particular research subfield. “A subfield-specific query (SubQ) approaches the same research question from a particular research subfield”
  • Term overlap: The degree to which a query and document share the same words or terms. “Lexical retrievers rank corpus papers through term overlap”
  • Tool-calling agent: A language-model agent that invokes external tools, such as retrievers, during reasoning. “The tool-calling agent follows the search-then-select structure for literature search”
  • Top-kk candidates: The highest-ranked kk documents returned by a retrieval system. “these agents still access the corpus through a retriever’s top-kk candidates at each step”
  • Undocumented relevance: Relevance judgments that are absent from the published paper and must be obtained from researchers directly. “SCHOLARCATALYST combines queries that precede a project's solution (Pre-discovery query), direct author annotation (Author annotation), and relevance judgment beyond what is published in the completed source paper (Undocumented relevance).”
  • Vector representation: A numerical representation of text used to calculate similarity for retrieval. “Embedding retrievers rank corpus papers through term overlap or learned semantic similarity”
  • Zero-shot dense retrieval: Dense retrieval performed without task-specific relevance labels or supervised adaptation. “Precise zero-shot dense retrieval without relevance labels”

Continue Learning

We haven't generated follow-up questions for this paper yet.

Tweets

Sign up for free to view the 16 tweets with 337 likes about this paper.