Abstract
Memory-based neural networks model temporal data by leveraging an ability toremember information for long periods. It is unclear, however, whether theyalso have an ability to perform complex relational reasoning with theinformation they remember. Here, we first confirm our intuitions that standardmemory architectures may struggle at tasks that heavily involve anunderstanding of the ways in which entities are connected -- i.e., tasksinvolving relational reasoning. We then improve upon these deficits by using anew memory module -- a \textit{Relational Memory Core} (RMC) -- which employsmulti-head dot product attention to allow memories to interact. Finally, wetest the RMC on a suite of tasks that may profit from more capable relationalreasoning across sequential information, and show large gains in RL domains(e.g. Mini PacMan), program evaluation, and language modeling, achievingstate-of-the-art results on the WikiText-103, Project Gutenberg, and GigaWorddatasets.