Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling

  • 2020-09-08 19:12:06
  • Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, Robert Frank
  • 3


By positing a relationship between naturalistic reading times andinformation-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008)provides a natural interface between language models and psycholinguisticmodels. This paper re-evaluates a claim due to Goodkind and Bicknell (2018)that a language model's ability to model reading times is a linear function ofits perplexity. By extending Goodkind and Bicknell's analysis to modern neuralarchitectures, we show that the proposed relation does not always hold for LongShort-Term Memory networks, Transformers, and pre-trained models. We introducean alternate measure of language modeling performance called predictabilitynorm correlation based on Cloze probabilities measured from human subjects. Ournew metric yields a more robust relationship between language model quality andpsycholinguistic modeling performance that allows for comparison between modelswith different training configurations.


Quick Read (beta)

loading the full paper ...