Abstract
Large language models (LLMs) have demonstrated remarkable capabilities acrossa range of natural language processing (NLP) tasks, capturing the attention ofboth practitioners and the broader public. A key question that now preoccupiesthe AI community concerns the capabilities and limitations of these models,with trustworthiness emerging as a central issue, particularly as LLMs areincreasingly applied in sensitive fields like healthcare and finance, whereerrors can have serious consequences. However, most previous studies on thetrustworthiness of LLMs have been limited to a single language, typically thepredominant one in the dataset, such as English. In response to the growingglobal deployment of LLMs, we introduce XTRUST, the first comprehensivemultilingual trustworthiness benchmark. XTRUST encompasses a diverse range oftopics, including illegal activities, hallucination, out-of-distribution (OOD)robustness, physical and mental health, toxicity, fairness, misinformation,privacy, and machine ethics, across 10 different languages. Using XTRUST, weconduct an empirical evaluation of the multilingual trustworthiness of fivewidely used LLMs, offering an in-depth analysis of their performance acrosslanguages and tasks. Our results indicate that many LLMs struggle with certainlow-resource languages, such as Arabic and Russian, highlighting theconsiderable room for improvement in the multilingual trustworthiness ofcurrent language models. The code is available athttps://github.com/LluckyYH/XTRUST.