Abstract
The emergence of large-scale pretrained language models has revolutionizedthe capabilities of new AI application, especially in the realm of craftingchatbots with distinct personas. Given the "stimulus-response" nature ofchatbots, this paper unveils an innovative open-ended interview-style approachfor personality assessment on role-playing chatbots, which offers a richercomprehension of their intrinsic personalities. We conduct personalityassessments on 32 role-playing chatbots created by the ChatHaruhi library,across both the Big Five and MBTI dimensions, and measure their alignment withhuman perception. Evaluation results underscore that modern role-playingchatbots based on LLMs can effectively portray personality traits ofcorresponding characters, with an alignment rate of 82.8% compared withhuman-perceived personalities. Besides, we also suggest potential strategiesfor shaping chatbots' personalities. Hence, this paper serves as a cornerstonestudy for role-playing chatbots that intersects computational linguistics andpsychology. Our resources are available athttps://github.com/LC1332/Chat-Haruhi-Suzumiya