Image outpainting seeks for a semantically consistent extension of the inputimage beyond its available content. Compared to inpainting -- filling inmissing pixels in a way coherent with the neighboring pixels -- outpainting canbe achieved in more diverse ways since the problem is less constrained by thesurrounding pixels. Existing image outpainting methods pose the problem as aconditional image-to-image translation task, often generating repetitivestructures and textures by replicating the content available in the inputimage. In this work, we formulate the problem from the perspective of invertinggenerative adversarial networks. Our generator renders micro-patchesconditioned on their joint latent code as well as their individual positions inthe image. To outpaint an image, we seek for multiple latent codes not onlyrecovering available patches but also synthesizing diverse outpainting bypatch-based generation. This leads to richer structure and content in theoutpainted regions. Furthermore, our formulation allows for outpaintingconditioned on the categorical input, thereby enabling flexible user controls.Extensive experimental results demonstrate the proposed method performsfavorably against existing in- and outpainting methods, featuring higher visualquality and diversity.