Abstract
In this work we integrate ideas from surface-based modeling with neuralsynthesis: we propose a combination of surface-based pose estimation and deepgenerative models that allows us to perform accurate pose transfer, i.e.synthesize a new image of a person based on a single image of that person andthe image of a pose donor. We use a dense pose estimation system that mapspixels from both images to a common surface-based coordinate system, allowingthe two images to be brought in correspondence with each other. We inpaint andrefine the source image intensities in the surface coordinate system, prior towarping them onto the target pose. These predictions are fused with those of aconvolutional predictive module through a neural synthesis module allowing fortraining the whole pipeline jointly end-to-end, optimizing a combination ofadversarial and perceptual losses. We show that dense pose estimation is asubstantially more powerful conditioning input than landmark-, or mask-basedalternatives, and report systematic improvements over state of the artgenerators on DeepFashion and MVC datasets.