Abstract
Pre-training by language modeling has become a popular and successfulapproach to NLP tasks, but we have yet to understand exactly what linguisticcapacities these pre-training processes confer upon models. In this paper weintroduce a suite of diagnostics drawn from human language experiments, whichallow us to ask targeted questions about the information used by languagemodels for generating predictions in context. As a case study, we apply thesediagnostics to the popular BERT model, finding that it can generallydistinguish good from bad completions involving shared category or rolereversal, albeit with less sensitivity than humans, and it robustly retrievesnoun hypernyms, but it struggles with challenging inferences and role-basedevent prediction -- and in particular, it shows clear insensitivity to thecontextual impacts of negation.