Abstract
As large scale vision language models become increasingly central to modernAI applications, understanding and mitigating social biases in these systemshas never been more critical. We investigate how dataset composition, modelsize, and multilingual training affect gender and racial bias in a popular VLM,CLIP, and its open source variants. In particular, we systematically evaluatemodels trained on varying dataset scales and architectures, as well asmultilingual versions encompassing English along with Persian, Turkish, andFinnish,languages with minimal gender marking. To assess social perceptionbias, we measure the zero-shot performance on face images featuring sociallycharged terms rooted in the psychological constructs of communion and agency,and demographic labeling bias using both the FairFace and PATA datasets. Our findings reveal three key insights. First, while larger training datasetscan mitigate some biases, they may also introduce or amplify others when thedata composition is imbalanced. Second, although increasing model sizegenerally improves performance, it does not consistently reduce bias and can,in certain cases, exacerbate it. Finally, while multilingual training broadenslinguistic coverage, it does not inherently neutralize bias and can transfer orintensify inequities across languages. Taken together, these results highlightthe necessity of inclusive, carefully curated training data to foster fairnessrather than relying solely on model scaling or language expansion. We provide asystematic evaluation for vision language bias across diverse demographics,underscoring the urgent need for intentional bias mitigation strategies innext-generation AI systems.