Abstract
We introduce three memory-augmented Recurrent Neural Networks (MARNNs) andexplore their capabilities on a series of simple language modeling tasks whosesolutions require stack-based mechanisms. We provide the first demonstration ofneural networks recognizing the generalized Dyck languages, which express thecore of what it means to be a language with hierarchical structure. Ourmemory-augmented architectures are easy to train in an end-to-end fashion andcan learn the Dyck languages over as many as six parenthesis-pairs, in additionto two deterministic palindrome languages and the string-reversal transductiontask, by emulating pushdown automata. Our experiments highlight the increasedmodeling capacity of memory-augmented models over simple RNNs, while inflectingour understanding of the limitations of these models.