Abstract
Language models have been supervised with both language-only objective andvisual grounding in existing studies of visual-grounded language learning.However, due to differences in the distribution and scale of visual-groundeddatasets and language corpora, the language model tends to mix up the contextof the tokens that occurred in the grounded data with those that do not. As aresult, during representation learning, there is a mismatch between the visualinformation and the contextual meaning of the sentence. To overcome thislimitation, we propose GroundedBERT - a grounded language learning method thatenhances the BERT representation with visually grounded information.GroundedBERT comprises two components: (i) the original BERT which captures thecontextual representation of words learned from the language corpora, and (ii)a visual grounding module which captures visual information learned fromvisual-grounded datasets. Moreover, we employ Optimal Transport (OT),specifically its partial variant, to solve the fractional alignment problembetween the two modalities. Our proposed method significantly outperforms thebaseline language models on various language tasks of the GLUE and SQuADdatasets.