Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training

  • 2018-10-10 03:23:38
  • Bei Liu, Jianlong Fu, Makoto P. Kato, Masatoshi Yoshikawa
  • 2

Abstract

Automatic generation of natural language from images has attracted extensiveattention. In this paper, we take one step further to investigate generation ofpoetic language (with multiple lines) to an image for automatic poetrycreation. This task involves multiple challenges, including discovering poeticclues from the image (e.g., hope from green), and generating poems to satisfyboth relevance to the image and poeticness in language level. To solve theabove challenges, we formulate the task of poem generation into two correlatedsub-tasks by multi-adversarial training via policy gradient, through which thecross-modal relevance and poetic language style can be ensured. To extractpoetic clues from images, we propose to learn a deep coupled visual-poeticembedding, in which the poetic representation from objects, sentiments andscenes in an image can be jointly learned. Two discriminative networks arefurther introduced to guide the poem generation, including a multi-modaldiscriminator and a poem-style discriminator. To facilitate the research, wehave released two poem datasets by human annotators with two distinctproperties: 1) the first human annotated image-to-poem pair dataset (with 8,292pairs in total), and 2) to-date the largest public English poem corpus dataset(with 92,265 different poems in total). Extensive experiments are conductedwith 8K images, among which 1.5K image are randomly picked for evaluation. Bothobjective and subjective evaluations show the superior performances against thestate-of-the-art methods for poem generation from images. Turing test carriedout with over 500 human subjects, among which 30 evaluators are poetry experts,demonstrates the effectiveness of our approach.

 

Quick Read (beta)

loading the full paper ...