Abstract
Commonsense reasoning is a long-standing challenge for deep learning. Forexample, it is difficult to use neural networks to tackle the Winograd Schemadataset~\cite{levesque2011winograd}. In this paper, we present a simple methodfor commonsense reasoning with neural networks, using unsupervised learning.Key to our method is the use of language models, trained on a massive amount ofunlabled data, to score multiple choice questions posed by commonsensereasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges,our models outperform previous state-of-the-art methods by a large margin,without using expensive annotated knowledge bases or hand-engineered features.We train an array of large RNN language models that operate at word orcharacter level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and acustomized corpus for this task and show that diversity of training data playsan important role in test performance. Further analysis also shows that oursystem successfully discovers important features of the context that decide thecorrect answer, indicating a good grasp of commonsense knowledge.