Abstract
Math reasoning is an active area of Large Language Model (LLM) researchbecause it is a hallmark of artificial intelligence and has implications inseveral domains, including math education. However, few works have explored howmath reasoning is encoded within LLM parameters and if it is a skill that canbe isolated within models. Doing so could allow targeted intervention toimprove math performance without altering non-math behavior and fosterunderstanding of how models encode math reasoning. We introduce MathNeurosurgery (MathNeuro), a computationally efficient method we use to isolatemath-specific parameters in LLMs using only forward passes. MathNeuro builds onexisting work by using weights and activations to calculate parameterimportance, but isolates math-specific parameters by filtering out thoseimportant for general language tasks. Through pruning parameters MathNeuroidentifies, we delete a LLM's math reasoning ability without significantlyimpacting its general language ability. Scaling the identified parameters by asmall constant improves a pretrained or instruction-tuned LLM's performance by4-17% on GSM8K and 5-35% on MATH while leaving non-math behavior unaltered.MathNeuro is also data efficient: most of its effectiveness holds whenidentifying math-specific parameters using a single sample. MathNeurohighlights the potential for future work to intervene on math-specificparameters.