Abstract
The limits of applicability of vision-and-language models are defined by thecoverage of their training data. Tasks like vision question answering (VQA)often require commonsense and factual information beyond what can be learnedfrom task-specific datasets. This paper investigates the injection of knowledgefrom general-purpose knowledge bases (KBs) into vision-and-languagetransformers. We use an auxiliary training objective that encourages thelearned representations to align with graph embeddings of matching entities ina KB. We empirically study the relevance of various KBs to multiple tasks andbenchmarks. The technique brings clear benefits to knowledge-demanding questionanswering tasks (OK-VQA, FVQA) by capturing semantic and relational knowledgeabsent from existing models. More surprisingly, the technique also benefitsvisual reasoning tasks (NLVR2, SNLI-VE). We perform probing experiments andshow that the injection of additional knowledge regularizes the space ofembeddings, which improves the representation of lexical and semanticsimilarities. The technique is model-agnostic and can expand the applicabilityof any vision-and-language transformer with minimal computational overhead.