Abstract
Conversational Recommendation Systems (CRSs) have recently started toleverage pretrained language models (LM) such as BERT for their ability tosemantically interpret a wide range of preference statement variations.However, pretrained LMs are well-known to be prone to intrinsic biases in theirtraining data, which may be exacerbated by biases embedded in domain-specificlanguage data(e.g., user reviews) used to fine-tune LMs for CRSs. We study arecently introduced LM-driven recommendation backbone (termed LMRec) of a CRSto investigate how unintended bias i.e., language variations such as namereferences or indirect indicators of sexual orientation or location that shouldnot affect recommendations manifests in significantly shifted price andcategory distributions of restaurant recommendations. The alarming results weobserve strongly indicate that LMRec has learned to reinforce harmfulstereotypes through its recommendations. For example, offhand mention of namesassociated with the black community significantly lowers the price distributionof recommended restaurants, while offhand mentions of common male-associatednames lead to an increase in recommended alcohol-serving establishments. Theseand many related results presented in this work raise a red flag that advancesin the language handling capability of LM-drivenCRSs do not come withoutsignificant challenges related to mitigating unintended bias in future deployedCRS assistants with a potential reach of hundreds of millions of end-users.