Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations

  • 2019-04-11 04:04:06
  • Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, Wei-Ying Ma
  • 25


We propose the Unified Visual-Semantic Embeddings (Unified VSE) for learninga joint space of visual representation and textual semantics. The model unifiesthe embeddings of concepts at different levels: objects, attributes, relations,and full scenes. We view the sentential semantics as a combination of differentsemantic components such as objects and relations; their embeddings are alignedwith different image regions. A contrastive learning approach is proposed forthe effective learning of this fine-grained alignment from only image-captionpairs. We also present a simple yet effective approach that enforces thecoverage of caption embeddings on the semantic components that appear in thesentence. We demonstrate that the Unified VSE outperforms baselines oncross-modal retrieval tasks; the enforcement of the semantic coverage improvesthe model's robustness in defending text-domain adversarial attacks. Moreover,our model empowers the use of visual cues to accurately resolve worddependencies in novel sentences.


Introduction (beta)



Conclusion (beta)