ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Abstract: What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper introduces ScholarCatalyst, a test for AI systems that search scientific papers.
The main question is:
Can an AI find the older research papers that might give scientists the key idea they need to solve a new problem?
Scientists do not only search for papers that use the same words as their question. Sometimes a very useful paper comes from a different area and contains an idea that can be adapted. The authors call these helpful papers “catalyst papers” because they can spark or speed up new research.
For example, imagine someone wants to build a better robot. The most useful paper might not be about robots at all. It might describe how ants organize themselves, giving the researcher a new idea.
2. What did the researchers want to find out?
The researchers focused on several questions:
- Can current AI search systems find papers that scientists say were useful or could have been useful?
- Are ordinary methods, such as matching similar words or meanings, good enough?
- Do more advanced AI “search agents” perform better than simpler search systems?
- Are the most inspiring papers usually the ones that look most similar to the research question?
- What kinds of ideas do catalyst papers provide?
The researchers were especially interested in early-stage research questions. These are questions written as they might have appeared before the researchers knew the final answer. This prevents the search system from simply looking for words from the finished paper.
3. How was the study carried out?
Building the benchmark
The team asked 184 researchers about 207 recent computer science and artificial intelligence projects.
For each project, the researchers:
- Described the project’s original research question.
- Wrote a broader question about the project’s main problem, called a CoreQ.
- Wrote more focused questions about particular research areas, called SubQs.
- Looked at possible earlier papers.
- Marked which papers had ideas that could help advance the project.
- Explained why each selected paper was useful.
The final benchmark contained:
- 894 research questions
- About 191,000 earlier papers
- Nearly 900 author-identified questions
- Helpful papers, called positive papers
- Related but unhelpful papers, called hard negatives
A hard negative is like a wrong answer that looks very convincing. For example, a paper might discuss almost exactly the same topic but offer no useful idea for the project.
Comparing different search systems
The researchers tested several kinds of systems:
- BM25, which mainly looks for matching words
- Embedding retrievers, which compare the meaning of the question and papers
- Scientific-paper search systems
- AI search agents that search repeatedly, break questions into smaller parts, and read results
- An advanced LLM called Claude Fable 5.1, used only as an extra comparison
The papers available to each system were limited to those published before the research project was completed. This made the test more realistic: the system could not simply use later papers that described the final discovery.
Measuring success
The researchers used Recall@20. This measures how many of the correct papers appeared in the system’s first 20 results.
For example, if the authors identified 10 useful papers and the search system found 5 of them in its top 20 results, its recall would be 50%.
4. What were the main findings?
Current systems miss many important papers
The strongest ordinary search systems found only about 48% of the author-identified catalyst papers in their top 20 results.
This means that more than half of the papers that experts considered useful were usually missing from the first 20 results.
Search was harder for the broad, central questions than for the more specific subfield questions.
More complicated AI agents did not solve the problem
The researchers expected AI search agents to do better because they can:
- Rewrite questions
- Search several times
- Break a question into smaller parts
- Read and compare papers
- Think about the results
However, these agents did not clearly beat simpler embedding search. One result found about 42% of the useful papers, while the simpler search found about 48%.
This happened because the agents still depended on a search tool to produce possible papers. If the search tool failed to show an important paper, the AI agent usually could not consider it.
This is similar to asking a student to choose the best book from a library shelf when the librarian has already placed the most useful books somewhere else. Even a very clever student cannot choose a book that was never put on the shelf.
Similarity is a weak clue
The useful papers were not always the papers that looked most similar to the question.
The authors found that rejected papers could be just as similar to the research question as the helpful papers. This means that matching words or general topics is not enough.
A catalyst paper might help because it:
- Introduces a method that can be adapted
- Provides evidence supporting a research direction
- Applies an idea in a new setting
- Shows a limitation that inspires a new approach
These useful connections can be difficult for a computer to notice because they may not be directly stated in the question.
Citations do not tell the whole story
The papers cited in a finished research article are not a complete record of what inspired the work.
For example:
- About 43.6% of useful papers for subfield questions were not cited in the final paper.
- Some useful papers came from outside the project’s main subject area.
- A paper can be important because it inspired an idea, even if the researchers did not directly use its method or cite it.
This shows why looking only at citation lists is not enough to understand how research develops.
Better access to the right papers matters more than reading more deeply
The study suggests that the biggest problem is not that AI cannot understand papers after finding them. The bigger problem is that AI often fails to find the right papers in the first place.
When the researchers gave an AI the project’s bibliography, its performance improved greatly because the bibliography already contained many of the useful papers.
However, allowing the AI to read longer sections of retrieved papers usually helped only a little. This supports the idea that finding the right candidates is the main challenge.
Even a newer model missed many useful papers
Claude Fable 5.1 was tested as a special comparison because it may have seen some of the papers during training. Even so, it found only about half of the useful papers in its top 20 results.
This suggests that simply making LLMs larger or newer may not be enough. They may need special training to recognize which ideas could help a new research project.
5. Why are these results important?
The paper shows that scientific research involves more than finding papers about the same topic.
A strong scientist must often make a creative connection:
“This paper is not directly about my problem, but its idea could help me solve it.”
Current AI systems are much better at finding papers with similar words or meanings than at making these deeper connections.
ScholarCatalyst is important because it provides a way to measure this ability. Instead of asking whether a paper is merely related to a topic, it asks whether experts believe the paper could actually help move a research project forward.
6. What could this mean for the future?
In the future, better scientific AI systems might help researchers by:
- Finding useful ideas from many different fields
- Suggesting papers that researchers would not normally discover
- Helping turn a vague question into a clearer research plan
- Revealing connections between areas of science that are usually separate
- Saving researchers time when searching through millions of papers
However, these systems need a kind of research intuition or “research taste.” They must learn to recognize valuable ideas even when the paper does not look obviously similar to the question.
The authors also point out that this could be especially useful because nearly half of the useful papers identified by researchers came from outside their main subject area. An AI system that can search across many fields could therefore discover ideas that no single human researcher has had time to read.
Simple conclusion
ScholarCatalyst shows that finding truly inspiring research papers is much harder than finding papers on the same topic. Current AI search systems can retrieve some useful papers, but they miss many others, and complicated search agents do not yet solve the problem.
The research provides a useful test for building better scientific AI. If future systems learn to recognize important connections between distant ideas, they could help scientists discover breakthroughs hidden among the enormous number of papers published every year.
Practical Applications
Immediate Applications
The paper’s findings support several applications that can be deployed with current retrieval infrastructure, provided they are treated as decision-support tools rather than autonomous scientific authorities.
- Research-literature discovery assistants for universities and R&D teams — Academia, software, industrial research
- Integrate the benchmark’s workflow into scholarly search systems: collect an early-stage research question, retrieve candidate papers with dense and lexical search, and present them with explanations, rationales, and confidence indicators.
- A practical workflow would combine
BM25and embedding retrieval, followed by human review of the top candidates. Because current systems achieve only approximately 39–51% Recall@20 depending on query type, results should be framed as possible catalysts, not comprehensive recommendations. - Dependencies: access to a sufficiently broad, temporally filtered corpus; high-quality metadata and abstracts; transparent ranking; and researcher verification.
- Early-stage project-scoping and grant-writing support — Academia, public-sector research, corporate R&D
- Researchers can use the CoreQ/SubQ distinction to separate:
- a broad question about the central research problem, and
- narrower searches from specific disciplinary perspectives.
- The resulting workflow can help identify methodological precedents, empirical evidence, limitations motivating new work, and ideas that may generalize across fields.
- This is particularly useful when preparing literature reviews, grant proposals, technical roadmaps, or project kickoffs.
- Dependencies: the user must provide a genuinely pre-discovery description of the problem and independently assess whether retrieved papers are actually applicable.
- Cross-disciplinary “adjacent field” search — Healthcare, robotics, energy, materials, finance, climate science
- Since nearly half of the credited catalyst papers came from outside the project’s immediate subject area, organizations can deliberately search neighboring disciplines rather than relying only on keyword-matched literature.
- For example, a robotics group might search computer vision, control theory, human-computer interaction, and neuroscience separately; an energy team might search optimization, materials science, economics, and power systems.
- Tools could generate several SubQs for different technical perspectives and merge the resulting candidate lists for expert review.
- Dependencies: broad indexing across disciplines, compatible terminology, and domain experts who can judge whether an analogy is technically meaningful rather than merely superficial.
- Human-in-the-loop literature triage — Libraries, research offices, information services
- The paper’s automated data-collection pipeline can be adapted into an annotation interface in which researchers:
- 1. describe an early-stage problem,
- 2. review candidate papers,
- 3. label useful and non-useful papers,
- 4. explain the reason for each judgment, and
- 5. refine the query.
- Libraries and research-support offices could use this process to create tailored reading lists for labs, research consortia, or emerging research areas.
- Dependencies: researcher time, incentives for annotation, privacy protection for unpublished research questions, and mechanisms for resolving disagreement between annotators.
- Evaluation of academic-search and scientific-agent products — Software and AI-platform development
- SCHOLARCATALYST can serve as an evaluation suite for comparing search engines, embedding models, rerankers, and research agents on a task that is harder than topical relevance.
- Product teams can measure
Recall@5,Recall@20,Recall@100, andnDCG@20, while separately tracking: - candidate coverage,
- ranking quality,
- citation versus non-citation discovery,
- cross-domain retrieval,
- and explanation quality.
- This can prevent systems from appearing effective merely because they return papers with similar vocabulary.
- Dependencies: licensing and availability of the benchmark, protection against training-data leakage, and evaluation on newly released instances.
- Temporal and reproducible literature search — Academia, regulated R&D, policy analysis
- The paper demonstrates a useful method for reconstructing the literature available before a project was completed. This can be used to create “as-of-date” literature reviews and audit trails.
- Potential applications include reproducible systematic reviews, patent and technology landscaping, historical analysis of research directions, and documentation of what information was available when a scientific or engineering decision was made.
- Dependencies: reliable publication dates, version control for preprints, deduplication of papers, and accurate bibliographic resolution.
- Improved researcher and student education — Education and daily academic life
- Graduate courses can use catalyst-retrieval exercises to teach students that important prior work may not be the most textually similar paper.
- Students can compare:
- papers returned by keyword or embedding search,
- papers identified by an expert,
- and their own explanations of why a paper could advance a project.
- This supports training in literature judgment, research taste, hypothesis formation, and critical reading.
- Dependencies: instructor moderation, access to suitable domain-specific corpora, and explicit warnings against treating model rankings as authoritative.
- Internal knowledge discovery in companies — Software engineering, pharmaceutical research, engineering
- Organizations can index internal technical reports, design documents, incident reports, patents, and external publications, then use the benchmark’s annotation approach to identify documents that could inform a new project.
- For example, an engineering team could search prior failure analyses for limitations that motivate a redesign, while a pharmaceutical team could identify methods from adjacent assay or modeling domains.
- Dependencies: secure retrieval over confidential data, access controls, document quality, and legal review for intellectual-property or regulatory constraints.
- Policy and funding-program design — Research policy and science administration
- Funding agencies can use author-annotated catalyst data to study how research progress is actually connected to prior work, including uncited or cross-disciplinary influences that bibliometric indicators miss.
- This could inform broader definitions of research impact, support interdisciplinary funding programs, and guide investments in open scholarly infrastructure.
- Dependencies: representative sampling beyond highly visible computer-science conference papers, safeguards against retrospective bias, and careful separation of discovery support from funding decisions.
Long-Term Applications
The following applications require substantial advances in candidate coverage, expert-level relevance modeling, data scale, or validation across disciplines.
- Expert-level scientific research agents — Academia, industrial R&D, healthcare, energy, robotics
- A mature agent could accept a half-formed research idea, decompose it into CoreQs and SubQs, search millions of papers, identify non-obvious catalyst papers, and explain the specific transferable insight in each one.
- Rather than simply producing a bibliography, it could organize results by inspiration type: method, evidence, limitation, generalization, or conceptual reframing.
- Such an agent could then propose experiments or project directions grounded in the retrieved literature.
- Dependencies: retrieval systems that overcome semantic-similarity limits; direct or richer access to full corpora; expert-validated training data; temporal controls; and reliable uncertainty estimation.
- Training models for “research taste” and inspiration-aware retrieval — AI research and software
- SCHOLARCATALYST can support new learning objectives in which models are trained to distinguish papers that are merely related from papers whose ideas could advance a project.
- Potential methods include:
- contrastive learning with author-approved positives and hard negatives,
- preference optimization over expert rationales,
- reinforcement learning from research-search trajectories,
- graph models combining papers, methods, citations, and concepts,
- and multi-stage retrieval followed by reasoning-based reranking.
- Dependencies: much larger and more diverse annotation sets, agreement studies, controls for hindsight bias, and protection against models memorizing completed source papers.
- Large-scale cross-domain discovery systems — Climate, medicine, materials, manufacturing, public health
- A system with broad expert-level retrieval could identify “undiscovered public knowledge”: useful combinations of methods or findings that exist in separate literatures but have not yet been connected.
- Examples include transferring machine-learning methods into medical imaging, adapting materials-discovery techniques to energy storage, or applying control methods from robotics to grid management.
- Dependencies: coverage of the full scholarly and technical literature, terminology normalization, domain-specific validation, intellectual-property review, and experimental confirmation of proposed transfers.
- Research-planning copilots that connect inspiration to execution — Laboratories and engineering organizations
- Future tools could transform retrieved catalyst papers into an actionable project plan:
- identify the transferable method,
- list required data or equipment,
- flag assumptions,
- predict likely limitations,
- suggest validation experiments,
- and maintain provenance links to the supporting literature.
- This would connect literature discovery with experiment design, simulation, code generation, and laboratory workflow management.
- Dependencies: accurate scientific reasoning, integration with laboratory and software systems, reproducibility of cited results, and human approval before experiments or operational changes.
- Automated technology and prior-art landscaping — Industry, patents, legal and policy analysis
- Inspiration-aware retrieval could supplement conventional citation and keyword searches in patent landscaping, competitive intelligence, standards analysis, and technology forecasting.
- It may uncover technically influential papers that are not highly cited, not explicitly referenced, or located outside the target industry’s usual vocabulary.
- Dependencies: legal-grade provenance, comprehensive patent and publication databases, strict distinction between scientific inspiration and legal prior art, and expert review by patent or regulatory professionals.
- Evidence-informed healthcare and biomedical research navigation — Healthcare and life sciences
- In the long term, biomedical researchers could use catalyst retrieval to find methods, limitations, or evidence from adjacent areas such as genomics, imaging, clinical trials, epidemiology, and computational biology.
- The system could help formulate translational research questions and identify promising methodological adaptations.
- It should not directly generate clinical recommendations; retrieved ideas would require laboratory, clinical, and regulatory validation.
- Dependencies: biomedical corpus coverage, patient-data privacy, reproducibility of findings, clinical expertise, safety review, and strict controls against unsupported medical inference.
- Policy intelligence for emerging scientific and technological risks — Government and international organizations
- Governments could use inspiration-aware systems to map how ideas move across fields and identify emerging research directions, dual-use concerns, or technology dependencies before they become visible through citation counts.
- Outputs could support horizon scanning, strategic investment, standards development, and research-security assessments.
- Dependencies: balanced global coverage, multilingual retrieval, protection against political or institutional bias, transparent evidence trails, and human-led interpretation.
- Personal research and learning companions — Daily life and lifelong education
- A future general-purpose assistant could help a learner move from a vague question to a structured exploration of relevant ideas, including papers from fields the learner would not normally encounter.
- For example, someone exploring battery recycling, educational technology, or personal data privacy could receive explanations of transferable concepts rather than only popular or keyword-matched sources.
- Dependencies: accessible explanations, reliable source attribution, protection from misinformation, adaptation to the user’s background, and clear communication that inspiration does not establish correctness or practical feasibility.
- Continually refreshed scientific benchmarks and model-audit infrastructure — Research governance and AI safety
- Because new papers can become benchmark instances before they enter model-training corpora, the pipeline could support rolling evaluations of scientific agents under changing knowledge conditions.
- This would help detect leakage, measure genuine retrieval progress, and evaluate whether agents improve in breadth, cross-domain transfer, and expert agreement.
- Dependencies: sustained author participation, standardized annotation protocols, secure temporal splits, regular corpus updates, and long-term institutional maintenance.
Glossary
- Agentic search: Information retrieval performed by an autonomous language-model agent that iteratively formulates queries and uses search tools. “Search agents use LLM reasoning to iteratively formulate queries and invoke these retrievers as tools over multiple turns”
- BM25: A probabilistic lexical-ranking algorithm that scores documents according to query-term occurrence and rarity. “We use Gemini 3.6. Flash (Google DeepMind, 2026b) reranks the pooled candidates for scientific relevance”
- Candidate coverage: The extent to which a retrieval system places relevant documents in the set available for subsequent ranking or reasoning. “We attribute this gap to candidate coverage, since regardless of how a model reasons, an agent can interact with the corpus only through its search queries.”
- Catalyst paper: A prior paper whose ideas could meaningfully inspire or advance a research project. “find its catalyst papers, by which we mean prior papers whose ideas did or could have meaningfully advanced the project”
- Citation graph: A network representing papers as nodes and their citation relationships as edges. “The resulting one-hop citation graph links inline citations to the referenced papers' metadata and abstracts.”
- Citation-intent annotation: A label describing the purpose or influence of one paper’s citation of another. “it derives positives from citation-intent annotations indicating methodological influence”
- Closed-book evaluation: An evaluation setting in which a model cannot access an external corpus or search tools. “Closed-book (no corpus)”
- Corpus: A collection of documents indexed for retrieval or analysis. “Our retrieval corpus contains 190,896 papers.”
- Dense retriever: A retrieval model that represents queries and documents as learned continuous vectors and compares them semantically. “Lexical retrievers and embedding retrievers rank corpus papers through term overlap or learned semantic similarity”
- Embedding space: A continuous vector space in which texts are positioned according to learned representations. “the embedding space itself does not place inspiration papers near their queries”
- Embedding retrieval: Retrieval based on semantic vector representations rather than exact word matching. “Agentic search does no better than embedding retrieval”
- Faceted query: A query that targets a particular aspect or dimension of a document or information need. “Existing scientific retrieval benchmarks evaluate how well systems find evidence for claims, retrieve documents for specified facets”
- Gold paper: A paper identified by the benchmark’s human annotators as relevant or inspirational. “Share of gold papers that each GPT-4.1 agent encounters during search”
- Gold injection: An experimental procedure that inserts missing relevant documents into a retrieval candidate set. “which replaces negative candidates with positive papers missing from the set while keeping the pool size fixed”
- Hard negative: A document that is related to a query but is judged irrelevant for the target task. “we also provide hard-negative documents, , which are related to but did not inspire the authors of .”
- HyDE: Hypothetical Document Embeddings, a retrieval technique that generates a hypothetical answer or document and embeds it to search for real documents. “both HyDE variants substantially reduce recall”
- Information retrieval: The study and practice of finding relevant information from a collection of documents. “Lexical retrievers rank corpus papers through term overlap or learned semantic similarity”
- Latent relevance: Relevance that is not explicitly stated in a query or document and must be inferred. “we pose an open-ended problem that requires inferring the latent scientific relevance of papers”
- Lexical retriever: A retrieval system that ranks documents using exact or statistical word overlap. “We include BM25 as a lexical baseline”
- Multi-vector retriever: A retriever that represents a document or query with multiple vectors rather than a single embedding. “We evaluate lexical, multi-vector, dense, and agentic retrieval.”
- nDCG@N: Normalized discounted cumulative gain measured at rank position , rewarding relevant documents placed near the top of a ranking. “normalized discounted cumulative gain (nDCG@N)”
- One-hop citation graph: A citation network containing the direct references of a paper, without recursively expanding to further citation levels. “The resulting one-hop citation graph links inline citations to the referenced papers' metadata and abstracts.”
- Oracle candidate pool: A candidate set assumed to contain the relevant documents, used to isolate ranking quality from initial retrieval quality. “We show later that reranking with an oracle candidate pool recovers part of this gap”
- Pre-discovery query: A query describing a research problem before its eventual solution or discovery is known. “SCHOLARCATALYST combines queries that precede a project's solution (Pre-discovery query)”
- Query expansion: The transformation of an original query into a longer or augmented query intended to improve retrieval. “We test whether an LLM can improve retrieval by reasoning over the original research question to generate single-query expansions”
- Query rewriting: Reformulating a query to improve the retrieval of relevant documents. “Query rewriting does not consistently improve retrieval.”
- Recall@N: The fraction of all relevant documents retrieved within the top results. “We evaluate retrieval performance using Recall@N, which measures the fraction of positive papers retrieved in the top results”
- Reranking: Reordering an initial set of retrieved documents using a more sophisticated relevance model. “Stronger rerankers benefit more from better candidate coverage.”
- Retrieval corpus: The document collection from which a retrieval system selects results. “By making the retrieval corpus contain only papers published before ”
- Retrieval cutoff: A temporal or model-knowledge boundary that limits which documents or information may be used. “Agent backbones must have a knowledge cutoff before every source paper”
- Semantic similarity: Similarity between texts based on their meanings or learned representations rather than literal word overlap. “semantic similarity as cosine similarity under Qwen3-Embedding-8B”
- Sparse retrieval: Retrieval using sparse representations, typically based on lexical features such as words or tokens. “Across sparse, dense, and multi-vector retrievers and LLM search agents”
- Subfield-specific query (SubQ): A query that frames a central research question from the perspective of a particular research subfield. “A subfield-specific query (SubQ) approaches the same research question from a particular research subfield”
- Term overlap: The degree to which a query and document share the same words or terms. “Lexical retrievers rank corpus papers through term overlap”
- Tool-calling agent: A language-model agent that invokes external tools, such as retrievers, during reasoning. “The tool-calling agent follows the search-then-select structure for literature search”
- Top- candidates: The highest-ranked documents returned by a retrieval system. “these agents still access the corpus through a retriever’s top- candidates at each step”
- Undocumented relevance: Relevance judgments that are absent from the published paper and must be obtained from researchers directly. “SCHOLARCATALYST combines queries that precede a project's solution (Pre-discovery query), direct author annotation (Author annotation), and relevance judgment beyond what is published in the completed source paper (Undocumented relevance).”
- Vector representation: A numerical representation of text used to calculate similarity for retrieval. “Embedding retrievers rank corpus papers through term overlap or learned semantic similarity”
- Zero-shot dense retrieval: Dense retrieval performed without task-specific relevance labels or supervised adaptation. “Precise zero-shot dense retrieval without relevance labels”