Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generation

  • 2019-08-01 07:58:50
  • Jing Wang, Yingwei Pan, Ting Yao, Jinhui Tang, Tao Mei
  • 29

Abstract

Image paragraph generation is the task of producing a coherent story (usuallya paragraph) that describes the visual content of an image. The problemnevertheless is not trivial especially when there are multiple descriptive anddiverse gists to be considered for paragraph generation, which often happens inreal images. A valid question is how to encapsulate such gists/topics that areworthy of mention from an image, and then describe the image from one topic toanother but holistically with a coherent structure. In this paper, we present anew design --- Convolutional Auto-Encoding (CAE) that purely employsconvolutional and deconvolutional auto-encoding framework for topic modeling onthe region-level features of an image. Furthermore, we propose an architecture,namely CAE plus Long Short-Term Memory (dubbed as CAE-LSTM), that novellyintegrates the learnt topics in support of paragraph generation. Technically,CAE-LSTM capitalizes on a two-level LSTM-based paragraph generation frameworkwith attention mechanism. The paragraph-level LSTM captures the inter-sentencedependency in a paragraph, while sentence-level LSTM is to generate onesentence which is conditioned on each learnt topic. Extensive experiments areconducted on Stanford image paragraph dataset, and superior results arereported when comparing to state-of-the-art approaches. More remarkably,CAE-LSTM increases CIDEr performance from 20.93% to 25.15%.

 

Quick Read (beta)

loading the full paper ...