Abstract
Deep learning sequence models have led to a marked increase in performancefor a range of Natural Language Processing tasks, but it remains an openquestion whether they are able to induce proper hierarchical generalizationsfor representing natural language from linear input alone. Work usingartificial languages as training input has shown that LSTMs are capable ofinducing the stack-like data structures required to represent context-free andcertain mildly context-sensitive languages---formal language classes whichcorrespond in theory to the hierarchical structures of natural language. Herewe present a suite of experiments probing whether neural language modelstrained on linguistic data induce these stack-like data structures and deploythem while incrementally predicting words. We study two natural languagephenomena: center embedding sentences and syntactic island constraints on thefiller--gap dependency. In order to properly predict words in these structures,a model must be able to temporarily suppress certain expectations and thenrecover those expectations later, essentially pushing and popping theseexpectations on a stack. Our results provide evidence that models cansuccessfully suppress and recover expectations in many cases, but do not fullyrecover their previous grammatical state.