Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics (2410.21272v1)

Abstract: Do LLMs solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types-such as neurons that activate when an operand falls within a certain range-and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a "bag of heuristics".

Arithmetic Without Algorithms: LLMs Solve Math with a Bag of Heuristics

This paper investigates whether LLMs solve arithmetic reasoning tasks using robust algorithms or if their behavior is primarily driven by memorization. By employing causal analysis, the authors identify and analyze circuits responsible for arithmetic calculations in various LLMs, discovering that arithmetic operations are primarily explained by a structured assembly of heuristic neurons, rather than coherent algorithms or mere memorization.

The research introduces a novel perspective by examining arithmetic reasoning at the level of individual neurons within identified circuits. Each neuron is found to implement simple heuristics, such as activating based on numerical patterns or operand ranges. These heuristics are aggregated into a "bag of heuristics" forming the operational structure used by LLMs to solve arithmetic tasks.

Key Findings and Methodology

1. Circuit Localization: The authors apply activation patching experiments across several LLMs, including Llama3-8B, Llama3-70B, Pythia-6.9B, and GPT-J, to identify model components (MLPs and attention heads) critical for arithmetic computations. These components collectively form what the authors define as arithmetic circuits.
2. Sparse Neuronal Contribution: Surprisingly, a sparse subset of neurons within these circuits suffices to predict arithmetic outcomes accurately. These neurons are distinct across different operations, indicating operator-specific heuristic implementations.
3. Neuron-Level Heuristics: By using the Logit Lens to project each neuron's value vectors onto numerical tokens, the paper reveals two prominent heuristic types: direct heuristics that directly enhance result tokens, and indirect heuristics that influence intermediate features. Neurons often activate when either operand or result meets specific numerical conditions, like congruence or range conformance.
4. Bag of Heuristics: The collective effect of multiple independent heuristic neurons is the core mechanism propelling LLMs' arithmetic logic. Experiments demonstrate that ablating neurons associated with a particular heuristic leads to a marked decrease in model accuracy. This correlation supports the assertion that arithmetic completion predominantly relies on combined heuristic effects rather than singular neuron responses.
5. Training Dynamics: The heuristics emerge early in training, suggesting no significant overhaul or replacement of initial arithmetic strategies with an advanced mechanism. This gradual development points to a learning trajectory that might contribute to over-specializing in heuristic-based solutions.

Implications and Future Directions

The findings challenge the notion of LLMs internalizing robust algorithmic processes, emphasizing a reliance on numerous heuristics. This insight complicates the landscape of interpretability, given that these heuristic combinations, albeit effective, may not generalize well to unseen or out-of-distribution data.

Practical implications include the potential need for new architectures or training paradigms that promote more generalized problem-solving methods over memorized heuristics. Theoretical implications extend to redefining how model interpretability frameworks account for multi-level abstraction interactions observed within LLMs.

Future research could explore regularization techniques to foster robust generalization or dissect whether similar heuristic formations are found in other reasoning tasks beyond arithmetic. Investigating alternative architectures that might naturally avoid heuristic over-reliance presents a compelling avenue for advancing LLM capabilities.

This paper offers a meticulous dissection of arithmetic reasoning in LLMs, paving the way for enhanced understanding and development of future AI models that may be capable of transcending heuristic dependencies.