Abstract
Recurrent neural networks empirically generate natural language with highsyntactic fidelity. However, their success is not well-understoodtheoretically. We provide theoretical insight into this success, proving in afinite-precision setting that RNNs can efficiently generate boundedhierarchical languages that reflect the scaffolding of natural language syntax.We introduce Dyck-($k$,$m$), the language of well-nested brackets (of $k$types) and $m$-bounded nesting depth, reflecting the bounded memory needs andlong-distance dependencies of natural language syntax. The best known resultsuse $O(k^{\frac{m}{2}})$ memory (hidden units) to generate these languages. Weprove that an RNN with $O(m \log k)$ hidden units suffices, an exponentialreduction in memory, by an explicit construction. Finally, we show that noalgorithm, even with unbounded computation, can suffice with $o(m \log k)$hidden units.