Abstract
Previous studies have found that PLM-based retrieval models exhibit apreference for LLM-generated content, assigning higher relevance scores tothese documents even when their semantic quality is comparable to human-writtenones. This phenomenon, known as source bias, threatens the sustainabledevelopment of the information access ecosystem. However, the underlying causesof source bias remain unexplored. In this paper, we explain the process ofinformation retrieval with a causal graph and discover that PLM-basedretrievers learn perplexity features for relevance estimation, causing sourcebias by ranking the documents with low perplexity higher. Theoretical analysisfurther reveals that the phenomenon stems from the positive correlation betweenthe gradients of the loss functions in language modeling task and retrievaltask. Based on the analysis, a causal-inspired inference-time debiasing methodis proposed, called Causal Diagnosis and Correction (CDC). CDC first diagnosesthe bias effect of the perplexity and then separates the bias effect from theoverall estimated relevance score. Experimental results across three domainsdemonstrate the superior debiasing effectiveness of CDC, emphasizing thevalidity of our proposed explanatory framework. Source codes are available athttps://github.com/WhyDwelledOnAi/Perplexity-Trap.