Extent of LLM performance attributable to reasoning vs memorized knowledge

Determine the extent to which large language models’ task performance reflects genuine reasoning as opposed to recall of memorized parametric world knowledge, by explicitly separating and measuring these contributions in controlled evaluations.

Background

The paper motivates SynthWorlds by noting that many evaluations confound genuine reasoning with memorized factual recall from pretraining. Because training data are massive and often undisclosed, benchmark scores may reflect parametric knowledge rather than reasoning ability. The authors aim to disentangle these effects via parallel real-mapped and synthetic-mapped corpora and tasks to quantify the “knowledge advantage gap.”

This open problem frames the need for methodologies that can isolate reasoning from memorization, enabling clearer scientific conclusions about LLMs’ capabilities and more reliable deployment in novel environments where prior memorized knowledge is less useful.