Learning event representations in image sequences by dynamic graph embedding

  • 2019-10-08 15:48:50
  • Mariella Dimiccoli, Herwig Wendt
  • 4

Abstract

Recently, self-supervised learning has proved to be effective to learnrepresentations of events in image sequences, where events are understood assets of temporally adjacent images that are semantically perceived as a whole.However, although this approach does not require expensive manual annotations,it is data hungry and suffers from domain adaptation problems. As analternative, in this work, we propose a novel approach for learning eventrepresentations named Dynamic Graph Embedding (DGE). The assumption underlyingour model is that a sequence of images can be represented by a graph thatencodes both semantic and temporal similarity. The key novelty of DGE is tolearn jointly the graph and its graph embedding. At its core, DGE works byiterating over two steps: 1) updating the graph representing the semantic andtemporal structure of the data based on the current data representation, and 2)updating the data representation to take into account the current data graphstructure. The main advantage of DGE over state-of-the-art self-supervisedapproaches is that it does not require any training set, but instead learnsiteratively from the data itself a low-dimensional embedding that reflectstheir temporal and semantic structure. Experimental results on two benchmarkdatasets of real image sequences captured at regular intervals demonstrate thatthe proposed DGE leads to effective event representations. In particular, itachieves robust temporal segmentation on the EDUBSeg and EDUBSeg-Desc benchmarkdatasets, outperforming the state of the art.

 

Quick Read (beta)

loading the full paper ...