Abstract
RNN language models have achieved state-of-the-art results on various tasks,but what exactly they are representing about syntax is as yet unclear. Here weinvestigate whether RNN language models learn humanlike word order preferencesin syntactic alternations. We collect language model surprisal scores forcontrolled sentence stimuli exhibiting major syntactic alternations in English:heavy NP shift, particle shift, the dative alternation, and the genitivealternation. We show that RNN language models reproduce human preferences inthese alternations based on NP length, animacy, and definiteness. We collecthuman acceptability ratings for our stimuli, in the first acceptabilityjudgment experiment directly manipulating the predictors of syntacticalternations. We show that the RNNs' performance is similar to the humanacceptability ratings and is not matched by an n-gram baseline model. Ourresults show that RNNs learn the abstract features of weight, animacy, anddefiniteness which underlie soft constraints on syntactic alternations.