Vision-to-Language Tasks Based on Attributes and Attention Mechanism

  • 2019-05-29 06:55:23
  • Xuelong Li, Aihong Yuan, Xiaoqiang Lu
  • 0

Abstract

Vision-to-language tasks aim to integrate computer vision and naturallanguage processing together, which has attracted the attention of manyresearchers. For typical approaches, they encode image into featurerepresentations and decode it into natural language sentences. While theyneglect high-level semantic concepts and subtle relationships between imageregions and natural language elements. To make full use of these information,this paper attempt to exploit the text guided attention and semantic-guidedattention (SA) to find the more correlated spatial information and reduce thesemantic gap between vision and language. Our method includes two levelattention networks. One is the text-guided attention network which is used toselect the text-related regions. The other is SA network which is used tohighlight the concept-related regions and the region-related concepts. At last,all these information are incorporated to generate captions or answers.Practically, image captioning and visual question answering experiments havebeen carried out, and the experimental results have shown the excellentperformance of the proposed approach.

 

Quick Read (beta)

loading the full paper ...