Abstract
To what extent can neural network models learn generalizations about languagestructure, and how do we find out what they have learned? We explore thesequestions by training neural models for a range of natural language processingtasks on a massively multilingual dataset of Bible translations in 1295languages. The learned language representations are then compared to existingtypological databases as well as to a novel set of quantitative syntactic andmorphological features obtained through annotation projection. We conclude thatsome generalizations are surprisingly close to traditional features fromlinguistic typology, but that most of our models, as well as those of previouswork, do not appear to have made linguistically meaningful generalizations.Careful attention to details in the evaluation turns out to be essential toavoid false positives. Furthermore, to encourage continued work in this field,we release several resources covering most or all of the languages in our data:(i) multiple sets of language representations, (ii) multilingual wordembeddings, (iii) projected and predicted syntactic and morphological features,(iv) software to provide linguistically sound evaluations of languagerepresentations.