Abstract
Role-playing has wide-ranging applications in customer support, embodiedagents, computational social science, etc. The influence of parametric worldknowledge of large language models (LLMs) often causes role-playing charactersto act out of character and hallucinate about things outside the scope of theirknowledge. In this work, we focus on the evaluation and mitigation ofhallucination in fictional character role-play. We introduce a dataset withmore than 2,000 characters and 72,000 interviews, including 18,000 adversarialquestions. We propose RoleFact, a role-playing method that mitigateshallucination by modulating the influence of parametric knowledge using apre-calibrated confidence threshold. Experiments show that the proposed methodimproves the factual precision of generated responses by 18% for adversarialquestions with a 44% reduction in temporal hallucination for time-sensitiveinterviews. The code and the dataset will be available athttps://github.com/NafisSadeq/rolefact.git.