Sem-GAN: Semantically-Consistent Image-to-Image Translation

  • 2018-07-12 02:55:19
  • Anoop Cherian, Alan Sullivan
  • 18

Abstract

Unpaired image-to-image translation is the problem of mapping an image in thesource domain to one in the target domain, without requiring correspondingimage pairs. To ensure the translated images are realistically plausible,recent works, such as Cycle-GAN, demands this mapping to be invertible. While,this requirement demonstrates promising results when the domains are unimodal,its performance is unpredictable in a multi-modal scenario such as in an imagesegmentation task. This is because, invertibility does not necessarily enforcesemantic correctness. To this end, we present a semantically-consistent GANframework, dubbed Sem-GAN, in which the semantics are defined by the classidentities of image segments in the source domain as produced by a semanticsegmentation algorithm. Our proposed framework includes consistency constraintson the translation task that, together with the GAN loss and thecycle-constraints, enforces that the images when translated will inherit theappearances of the target domain, while (approximately) maintaining theiridentities from the source domain. We present experiments on severalimage-to-image translation tasks and demonstrate that Sem-GAN improves thequality of the translated images significantly, sometimes by more than 20% onthe FCN score. Further, we show that semantic segmentation models, trained withsynthetic images translated via Sem-GAN, leads to significantly bettersegmentation results than other variants.

 

Quick Read (beta)

loading the full paper ...