Abstract
Do large language models (LLMs) solve reasoning tasks by learning robustgeneralizable algorithms, or do they memorize training data? To investigatethis question, we use arithmetic reasoning as a representative task. Usingcausal analysis, we identify a subset of the model (a circuit) that explainsmost of the model's behavior for basic arithmetic logic and examine itsfunctionality. By zooming in on the level of individual circuit neurons, wediscover a sparse set of important neurons that implement simple heuristics.Each heuristic identifies a numerical input pattern and outputs correspondinganswers. We hypothesize that the combination of these heuristic neurons is themechanism used to produce correct arithmetic answers. To test this, wecategorize each neuron into several heuristic types-such as neurons thatactivate when an operand falls within a certain range-and find that theunordered combination of these heuristic types is the mechanism that explainsmost of the model's accuracy on arithmetic prompts. Finally, we demonstratethat this mechanism appears as the main source of arithmetic accuracy early intraining. Overall, our experimental results across several LLMs show that LLMsperform arithmetic using neither robust algorithms nor memorization; rather,they rely on a "bag of heuristics".