Abstract
Previous researches on dialogue system assessment usually focus on thequality evaluation (e.g. fluency, relevance, etc) of responses generated by thechatbots, which are local and technical metrics. For a chatbot which respondsto millions of online users including minors, we argue that it should have ahealthy mental tendency in order to avoid the negative psychological impact onthem. In this paper, we establish several mental health assessment dimensionsfor chatbots (depression, anxiety, alcohol addiction, empathy) and introducethe questionnaire-based mental health assessment methods. We conductassessments on some well-known open-domain chatbots and find that there aresevere mental health issues for all these chatbots. We consider that it is dueto the neglect of the mental health risks during the dataset building and themodel training procedures. We expect to attract researchers' attention to theserious mental health problems of chatbots and improve the chatbots' ability inpositive emotional interaction.