Reasoning versus Recall as the Source of LLM Success

Determine whether the observed success of large language models (LLMs) across diverse tasks primarily reflects genuine conceptual reasoning ability or instead arises from sophisticated associative recall of memorized information, in order to clarify the nature of their problem-solving competence.

Background

The paper introduces AInstein, a framework designed to evaluate whether LLMs can generate valid solutions to AI research problems using only their pretrained parametric knowledge, explicitly avoiding domain-specific fine-tuning and retrieval augmentation. This setup aims to disentangle genuine reasoning from mere associative recall by extracting generalized problem statements from ICLR abstracts and having solver agents propose and refine solutions through iterative critique loops.

The central motivation is to test if LLMs can act as autonomous scientific problem-solvers rather than pattern matchers. Metrics such as Success Rate, Rediscovery, and Novelty are used to assess whether proposed solutions are valid, align with human methods, or present original alternatives, thereby addressing the uncertainty regarding the source of LLMs' apparent capabilities.