Abstract

Retrieval-augmented language models pose a promising alternative to standardlanguage modeling. During pretraining, these models search in a corpus ofdocuments for contextually relevant information that could aid the languagemodeling objective. We introduce an 'ideal retrieval' methodology to studythese models in a fully controllable setting. We conduct an extensiveevaluation to examine how retrieval augmentation affects the behavior of theunderlying language model. Among other things, we observe that these models: i)save substantially less world knowledge in their weights, ii) are better atunderstanding local context and inter-word dependencies, but iii) are worse atcomprehending global context.

 

Quick Read (beta)