Abstract
Text-to-Image (T2I) models are being increasingly adopted in diverse globalcommunities where they create visual representations of their unique cultures.Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realismof generated images, overlooking the critical dimension of cultural competence.In this work, we introduce a framework to evaluate cultural competence of T2Imodels along two crucial dimensions: cultural awareness and cultural diversity,and present a scalable approach using a combination of structured knowledgebases and large language models to build a large dataset of cultural artifactsto enable this evaluation. In particular, we apply this approach to build CUBE(CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark toevaluate cultural competence of T2I models. CUBE covers cultural artifactsassociated with 8 countries across different geo-cultural regions and along 3concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set ofhigh-quality prompts that enable the evaluation of cultural awareness, and 2)CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding toevaluate cultural diversity. We also introduce cultural diversity as a novelT2I evaluation component, leveraging quality-weighted Vendi score. Ourevaluations reveal significant gaps in the cultural awareness of existingmodels across countries and provide valuable insights into the culturaldiversity of T2I outputs for under-specified prompts. Our methodology isextendable to other cultural regions and concepts, and can facilitate thedevelopment of T2I models that better cater to the global population.