By positing a relationship between naturalistic reading times andinformation-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008)provides a natural interface between language models and psycholinguisticmodels. This paper re-evaluates a claim due to Goodkind and Bicknell (2018)that a language model's ability to model reading times is a linear function ofits perplexity. By extending Goodkind and Bicknell's analysis to modern neuralarchitectures, we show that the proposed relation does not always hold for LongShort-Term Memory networks, Transformers, and pre-trained models. We introducean alternate measure of language modeling performance called predictabilitynorm correlation based on Cloze probabilities measured from human subjects. Ournew metric yields a more robust relationship between language model quality andpsycholinguistic modeling performance that allows for comparison between modelswith different training configurations.