Abstract
There is an ongoing debate in the NLP community whether modern languagemodels contain linguistic knowledge, recovered through so-called probes. Inthis paper, we study whether linguistic knowledge is a necessary condition forthe good performance of modern language models, which we call the\textit{rediscovery hypothesis}. In the first place, we show that languagemodels that are significantly compressed but perform well on their pretrainingobjectives retain good scores when probed for linguistic structures. Thisresult supports the rediscovery hypothesis and leads to the second contributionof our paper: an information-theoretic framework that relates language modelingobjectives with linguistic information. This framework also provides a metricto measure the impact of linguistic information on the word prediction task. Wereinforce our analytical results with various experiments, both on syntheticand on real NLP tasks in English.