Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter

  • 2020-07-14 17:57:59
  • Guilin Liu, Rohan Taori, Ting-Chun Wang, Zhiding Yu, Shiqiu Liu, Fitsum A. Reda, Karan Sapra, Andrew Tao, Bryan Catanzaro
  • 17

Abstract

Conventional CNNs for texture synthesis consist of a sequence of(de)-convolution and up/down-sampling layers, where each layer operates locallyand lacks the ability to capture the long-term structural dependency requiredby texture synthesis. Thus, they often simply enlarge the input texture, ratherthan perform reasonable synthesis. As a compromise, many recent methodssacrifice generalizability by training and testing on the same single (or fixedset of) texture image(s), resulting in huge re-training time costs for unseenimages. In this work, based on the discovery that the assembling/stitchingoperation in traditional texture synthesis is analogous to a transposedconvolution operation, we propose a novel way of using transposed convolutionoperation. Specifically, we directly treat the whole encoded feature map of theinput texture as transposed convolution filters and the features'self-similarity map, which captures the auto-correlation information, as inputto the transposed convolution. Such a design allows our framework, oncetrained, to be generalizable to perform synthesis of unseen textures with asingle forward pass in nearly real-time. Our method achieves state-of-the-arttexture synthesis quality based on various metrics. While self-similarity helpspreserve the input textures' regular structural patterns, our framework canalso take random noise maps for irregular input textures instead ofself-similarity maps as transposed convolution inputs. It allows to get morediverse results as well as generate arbitrarily large texture outputs bydirectly sampling large noise maps in a single pass as well.

 

Quick Read (beta)

loading the full paper ...