Abstract
Recent work has shown that inducing a large language model (LLM) to generateexplanations prior to outputting an answer is an effective strategy to improveperformance on a wide range of reasoning tasks. In this work, we show thatneural rankers also benefit from explanations. We use LLMs such as GPT-3.5 toaugment retrieval datasets with explanations and train a sequence-to-sequenceranking model to output a relevance label and an explanation for a givenquery-document pair. Our model, dubbed ExaRanker, finetuned on a few thousandexamples with synthetic explanations performs on par with models finetuned on3x more examples without explanations. Furthermore, the ExaRanker model incursno additional computational cost during ranking and allows explanations to berequested on demand.