Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting

  • 2020-05-19 18:55:32
  • Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, Zhan Xu
  • 32

Abstract

Recently data-driven image inpainting methods have made inspiring progress,impacting fundamental image editing tasks such as object removal and damagedimage repairing. These methods are more effective than classic approaches,however, due to memory limitations they can only handle low-resolution inputs,typically smaller than 1K. Meanwhile, the resolution of photos captured withmobile devices increases up to 8K. Naive up-sampling of the low-resolutioninpainted result can merely yield a large yet blurry result. Whereas, adding ahigh-frequency residual image onto the large blurry image can generate a sharpresult, rich in details and textures. Motivated by this, we propose aContextual Residual Aggregation (CRA) mechanism that can produce high-frequencyresiduals for missing contents by weighted aggregating residuals fromcontextual patches, thus only requiring a low-resolution prediction from thenetwork. Since convolutional layers of the neural network only need to operateon low-resolution inputs and outputs, the cost of memory and computing power isthus well suppressed. Moreover, the need for high-resolution training datasetsis alleviated. In our experiments, we train the proposed model on small imageswith resolutions 512x512 and perform inference on high-resolution images,achieving compelling inpainting quality. Our model can inpaint images as largeas 8K with considerable hole sizes, which is intractable with previouslearning-based approaches. We further elaborate on the light-weight design ofthe network architecture, achieving real-time performance on 2K images on a GTX1080 Ti GPU. Codes are available at: Atlas200dk/sample-imageinpainting-HiFill.

 

Quick Read (beta)

loading the full paper ...