Abstract
We generalize the notion of social biases from language embeddings togrounded vision and language embeddings. Biases are present in groundedembeddings, and indeed seem to be equally or more significant than forungrounded embeddings. This is despite the fact that vision and language cansuffer from different biases, which one might hope could attenuate the biasesin both. Multiple ways exist to generalize metrics measuring bias in wordembeddings to this new setting. We introduce the space of generalizations(Grounded-WEAT and Grounded-SEAT) and demonstrate that three generalizationsanswer different yet important questions about how biases, language, and visioninteract. These metrics are used on a new dataset, the first for grounded bias,created by augmenting extending standard linguistic bias benchmarks with 10,228images from COCO, Conceptual Captions, and Google Images. Dataset constructionis challenging because vision datasets are themselves very biased. The presenceof these biases in systems will begin to have real-world consequences as theyare deployed, making carefully measuring bias and then mitigating it criticalto building a fair society.