Abstract
We propose transfer learning as a method for analyzing the encoding ofgrammatical structure in neural language models. We train LSTMs onnon-linguistic data and evaluate their performance on natural language toassess which kinds of data induce generalizable structural features that LSTMscan use for natural language. We find that training on non-linguistic data withlatent structure (MIDI music or Java code) improves test performance on naturallanguage, despite no overlap in surface form or vocabulary. Training onartificial languages containing recursion (hierarchical structure) alsoimproves performance on natural language, again with no vocabulary overlap.Surprisingly, training on artificial languages consisting of sets of separatedpairs of words, but with no recursion, improves performance on natural languageas well as recursive languages do. Experiments on transfer between naturallanguages show that zero-shot performance on a test language is highlycorrelated with typological syntactic similarity to the training language,suggesting that representations induced from natural languages correspond tothe cross-linguistic syntactic properties studied in linguistic typology. Ourresults provide insights into the ways that neural models represent abstractsyntactic structure, and also about the kind of structural inductive biaseswhich a learner needs to model language.