Abstract
Neural networks can learn to represent and manipulate numerical information,but they seldom generalize well outside of the range of numerical valuesencountered during training. To encourage more systematic numericalextrapolation, we propose an architecture that represents numerical quantitiesas linear activations which are manipulated using primitive arithmeticoperators, controlled by learned gates. We call this module a neural arithmeticlogic unit (NALU), by analogy to the arithmetic logic unit in traditionalprocessors. Experiments show that NALU-enhanced neural networks can learn totrack time, perform arithmetic over images of numbers, translate numericallanguage into real-valued scalars, execute computer code, and count objects inimages. In contrast to conventional architectures, we obtain substantiallybetter generalization both inside and outside of the range of numerical valuesencountered during training, often extrapolating orders of magnitude beyondtrained numerical ranges.