Reliable suggestion of new scientific ideas and impact evaluation by large-language models

Determine how large-language models—specifically GPT-4, Gemini, and LLaMA-2—can reliably suggest new scientific ideas and evaluate the prospective impact of those ideas in the near term.

Background

The paper positions large-LLMs such as GPT-4, Gemini, and LLaMA-2 as a natural first choice for AI assistants in scientific discovery but highlights limitations in their scientific reasoning. The authors argue that despite the advances in LLMs, their capability to generate novel scientific ideas and assess their potential impact remains uncertain.

As an alternative, the paper develops an evolving, citation-augmented knowledge graph and demonstrates a machine-learning approach that forecasts the impact of new, previously unstudied concept pairs. This open question concerning LLMs motivates the need for complementary methods to assist researchers at the ideation stage.