Representation of linguistic phenomena in computational language models istypically assessed against the predictions of existing linguistic theories ofthese phenomena. Using the notion of polarity as a case study, we show thatthis is not always the most adequate set-up. We probe polarity via so-called'negative polarity items' (in particular, English 'any') in two pre-trainedTransformer-based models (BERT and GPT-2). We show that -- at least forpolarity -- metrics derived from language models are more consistent with datafrom psycholinguistic experiments than linguistic theory predictions.Establishing this allows us to more adequately evaluate the performance oflanguage models and also to use language models to discover new insights intonatural language grammar beyond existing linguistic theories. Overall, ourresults encourage a closer tie between experiments with human subjects and withlanguage models. We propose methods to enable this closer tie, with languagemodels as part of experimental pipeline, and show this pipeline at work.