Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward

  • 2018-02-13 19:34:43
  • Kaiyang Zhou, Yu Qiao, Tao Xiang
  • 1

Abstract

Video summarization aims to facilitate large-scale video browsing byproducing short, concise summaries that are diverse and representative oforiginal videos. In this paper, we formulate video summarization as asequential decision-making process and develop a deep summarization network(DSN) to summarize videos. DSN predicts for each video frame a probability,which indicates how likely a frame is selected, and then takes actions based onthe probability distributions to select frames, forming video summaries. Totrain our DSN, we propose an end-to-end, reinforcement learning-basedframework, where we design a novel reward function that jointly accounts fordiversity and representativeness of generated summaries and does not rely onlabels or user interactions at all. During training, the reward function judgeshow diverse and representative the generated summaries are, while DSN strivesfor earning higher rewards by learning to produce more diverse and morerepresentative summaries. Since labels are not required, our method can befully unsupervised. Extensive experiments on two benchmark datasets show thatour unsupervised method not only outperforms other state-of-the-artunsupervised methods, but also is comparable to or even superior than most ofpublished supervised approaches.

 

Quick Read (beta)

loading the full paper ...