Abstract
Mathematical reasoning---a core ability within human intelligence---presentssome unique challenges as a domain: we do not come to understand and solvemathematical problems primarily on the back of experience and evidence, but onthe basis of inferring, learning, and exploiting laws, axioms, and symbolmanipulation rules. In this paper, we present a new challenge for theevaluation (and eventually the design) of neural architectures and similarsystem, developing a task suite of mathematics problems involving sequentialquestions and answers in a free-form textual input/output format. Thestructured nature of the mathematics domain, covering arithmetic, algebra,probability and calculus, enables the construction of training and test splitsdesigned to clearly illuminate the capabilities and failure-modes of differentarchitectures, as well as evaluate their ability to compose and relateknowledge and learned processes. Having described the data generation processand its potential future expansions, we conduct a comprehensive analysis ofmodels from two broad classes of the most powerful sequence-to-sequencearchitectures and find notable differences in their ability to resolvemathematical problems and generalize their knowledge.