Abstract
Native speakers can judge whether a sentence is an acceptable instance oftheir language. Acceptability provides a means of evaluating whethercomputational language models are processing language in a human-like manner.We test the ability of computational language models, simple language features,and word embeddings to predict native English speakers judgments ofacceptability on English-language essays written by non-native speakers. Wefind that much of the sentence acceptability variance can be captured by acombination of features including misspellings, word order, and word similarity(Pearson's r = 0.494). While predictive neural models fit acceptabilityjudgments well (r = 0.527), we find that a 4-gram model with statisticalsmoothing is just as good (r = 0.528). Thanks to incorporating a count ofmisspellings, our 4-gram model surpasses both the previous unsupervisedstate-of-the art (Lau et al., 2015; r = 0.472), and the average non-expertnative speaker (r = 0.46). Our results demonstrate that acceptability is wellcaptured by n-gram statistics and simple language features.