Abstract
Does vision-and-language (VL) training change the linguistic representationsof language models in meaningful ways? Most results in the literature haveshown inconsistent or marginal differences, both behaviorally andrepresentationally. In this work, we start from the hypothesis that the domainin which VL training could have a significant effect is lexical-conceptualknowledge, in particular its taxonomic organization. Through comparing minimalpairs of text-only LMs and their VL-trained counterparts, we first show thatthe VL models often outperform their text-only counterparts on a text-onlyquestion-answering task that requires taxonomic understanding of conceptsmentioned in the questions. Using an array of targeted behavioral andrepresentational analyses, we show that the LMs and VLMs do not differsignificantly in terms of their taxonomic knowledge itself, but they differ inhow they represent questions that contain concepts in a taxonomic relation vs.a non-taxonomic relation. This implies that the taxonomic knowledge itself doesnot change substantially through additional VL training, but VL training doesimprove the deployment of this knowledge in the context of a specific task,even when the presentation of the task is purely linguistic.