Abstract
Large language models have shown impressive few-shot results on a wide rangeof tasks. However, when knowledge is key for such results, as is the case fortasks such as question answering and fact checking, massive parameter counts tostore knowledge seem to be needed. Retrieval augmented models are known toexcel at knowledge intensive tasks without the need for as many parameters, butit is unclear whether they work in few-shot settings. In this work we presentAtlas, a carefully designed and pre-trained retrieval augmented language modelable to learn knowledge intensive tasks with very few training examples. Weperform evaluations on a wide range of tasks, including MMLU, KILT andNaturalQuestions, and study the impact of the content of the document index,showing that it can easily be updated. Notably, Atlas reaches over 42\%accuracy on Natural Questions using only 64 examples, outperforming a 540Bparameters model by 3% despite having 50x fewer parameters.