Abstract

Rerankers, typically cross-encoders, are often used to re-score the documentsretrieved by cheaper initial IR systems. This is because, though expensive,rerankers are assumed to be more effective. We challenge this assumption bymeasuring reranker performance for full retrieval, not just re-scoringfirst-stage retrieval. Our experiments reveal a surprising trend: the bestexisting rerankers provide diminishing returns when scoring progressively moredocuments and actually degrade quality beyond a certain limit. In fact, in thissetting, rerankers can frequently assign high scores to documents with nolexical or semantic overlap with the query. We hope that our findings will spurfuture research to improve reranking.

 

Quick Read (beta)