Quantifying memorization versus generalized reasoning in LLM mathematical problem solving

Determine the proportion of apparent mathematical reasoning exhibited by large language models that is attributable to memorization of training data and shallow heuristics, as opposed to learned general principles of reasoning that generalize beyond the training examples when solving high-school-level word problems.

Background

LLMs have been observed to produce seemingly non-trivial reasoning, especially under chain-of-thought prompting, despite being primarily trained for next-token prediction. This raises a fundamental question about the mechanisms underpinning their performance on mathematical word problems.

The paper evaluates eight contemporary LLMs on 50 newly constructed high-school-level problems and analyzes both answers and solution steps. While models demonstrate broad mathematical knowledge, they also show systematic reasoning failures. Despite these empirical observations, the underlying balance between memorization and genuine generalization remains unresolved and is explicitly noted by the authors as unclear.