Abstract
As large-scale vision-language models (VLMs) become increasingly central tomodern AI applications, understanding and mitigating social biases in thesesystems has never been more critical.We investigate how dataset composition,model size, and multilingual training affect gender and racial bias in apopular VLM, CLIP, and its open-source variants. In particular, wesystematically evaluate models trained on varying dataset scales andarchitectures, as well as multilingual versions encompassing English along withPersian, Turkish, and Finnish, languages with minimal gender marking. To assesssocial perception bias, we measure the zero-shot performance on face imagesfeaturing socially charged terms rooted in the psychological constructs ofcommunion and agency, and demographic labeling bias using both the FairFace andPATA datasets. Our findings reveal three key insights. First, while larger training datasetscan mitigate some biases, they may also introduce or amplify others when thedata composition is imbalanced. Second, although increasing model sizegenerally improves performance, it does not consistently reduce bias and can,in certain cases, exacerbate it. Finally, while multilingual training broadenslinguistic coverage, it does not inherently neutralize bias and can transfer orintensify inequities across languages. Taken together, these results highlightthe necessity of inclusive, carefully curated training data to foster fairnessrather than relying solely on model scaling or language expansion. We provide asystematic evaluation of vision language bias across diverse demographics,underscoring the urgent need for intentional bias mitigation strategies in nextgeneration AI systems.