Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics

  • 2022-07-31 18:36:09
  • Xiaoyuan Guo, Jiali Duan, C. -C. Jay Kuo, Judy Wawira Gichoya, Imon Banerjee
  • 0


Language modality within the vision language pretraining framework isinnately discretized, endowing each word in the language vocabulary a semanticmeaning. In contrast, visual modality is inherently continuous andhigh-dimensional, which potentially prohibits the alignment as well as fusionbetween vision and language modalities. We therefore propose to "discretize"the visual representation by joint learning a codebook that imbues each visualtoken a semantic. We then utilize these discretized visual semantics asself-supervised ground-truths for building our Masked Image Modeling objective,a counterpart of Masked Language Modeling which proves successful for languagemodels. To optimize the codebook, we extend the formulation of VQ-VAE whichgives a theoretic guarantee. Experiments validate the effectiveness of ourapproach across common vision-language benchmarks.


Quick Read (beta)

loading the full paper ...