Abstract
We investigate the internal representations of vision-and-language models(VLMs) and how they encode task representations. We consider tasks specifiedthrough examples or instructions, using either text or image inputs.Surprisingly, we find that conceptually similar tasks are mapped to similartask vector representations, regardless of how they are specified. Our findingssuggest that to output answers, tokens in VLMs undergo three distinct phases:input, task, and answer, a process which is consistent across differentmodalities and specifications. The task vectors we identify in VLMs are generalenough to be derived in one modality (e.g., text) and transferred to another(e.g., image). Additionally, we find that ensembling exemplar and instructionbased task vectors produce better task representations. Taken together, theseinsights shed light on the underlying mechanisms of VLMs, particularly theirability to represent tasks in a shared manner across different modalities andtask specifications. Project page:https://task-vectors-are-cross-modal.github.io.