Distilling Knowledge from Reader to Retriever for Question Answering

  • 2022-08-04 18:36:08
  • Gautier Izacard, Edouard Grave
  • 0

Abstract

The task of information retrieval is an important component of many naturallanguage processing systems, such as open domain question answering. Whiletraditional methods were based on hand-crafted features, continuousrepresentations based on neural networks recently obtained competitive results.A challenge of using such methods is to obtain supervised data to train theretriever model, corresponding to pairs of query and support documents. In thispaper, we propose a technique to learn retriever models for downstream tasks,inspired by knowledge distillation, and which does not require annotated pairsof query and documents. Our approach leverages attention scores of a readermodel, used to solve the task based on retrieved documents, to obtain syntheticlabels for the retriever. We evaluate our method on question answering,obtaining state-of-the-art results.

 

Quick Read (beta)

loading the full paper ...