Abstract
Transformer based methods have achieved great success in image inpaintingrecently. However, we find that these solutions regard each pixel as a token,thus suffering from an information loss issue from two aspects: 1) Theydownsample the input image into much lower resolutions for efficiencyconsideration. 2) They quantize $256^3$ RGB values to a small number (such as512) of quantized color values. The indices of quantized pixels are used astokens for the inputs and prediction targets of the transformer. To mitigatethese issues, we propose a new transformer based framework called "PUT".Specifically, to avoid input downsampling while maintaining computationefficiency, we design a patch-based auto-encoder P-VQVAE. The encoder convertsthe masked image into non-overlapped patch tokens and the decoder recovers themasked regions from the inpainted tokens while keeping the unmasked regionsunchanged. To eliminate the information loss caused by input quantization, anUn-quantized Transformer is applied. It directly takes features from theP-VQVAE encoder as input without any quantization and only regards thequantized tokens as prediction targets. Furthermore, to make the inpaintingprocess more controllable, we introduce semantic and structural conditions asextra guidance. Extensive experiments show that our method greatly outperformsexisting transformer based methods on image fidelity and achieves much higherdiversity and better fidelity than state-of-the-art pluralistic inpaintingmethods on complex large-scale datasets (e.g., ImageNet). Codes are availableat https://github.com/liuqk3/PUT.