Abstract
Patch-based methods and deep networks have been employed to tackle imageinpainting problem, with their own strengths and weaknesses. Patch-basedmethods are capable of restoring a missing region with high-quality texturethrough searching nearest neighbor patches from the unmasked regions. However,these methods bring problematic contents when recovering large missing regions.Deep networks, on the other hand, show promising results in completing largeregions. Nonetheless, the results often lack faithful and sharp details thatresemble the surrounding area. By bringing together the best of both paradigms,we propose a new deep inpainting framework where texture generation is guidedby a texture memory of patch samples extracted from unmasked regions. Theframework has a novel design that allows texture memory retrieval to be trainedend-to-end with the deep inpainting network. In addition, we introduce a patchdistribution loss to encourage high-quality patch synthesis. The proposedmethod shows superior performance both qualitatively and quantitatively onthree challenging image benchmarks, i.e., Places, CelebA-HQ, and ParisStreet-View datasets.