Generative Adversarial Networks (GANs) have recently advanced image synthesisby learning the underlying distribution of observed data in an unsupervisedmanner. However, how the features trained from solving the task of imagesynthesis are applicable to visual tasks remains seldom explored. In this work,we show that learning to synthesize images is able to bring remarkablehierarchical visual features that are generalizable across a wide range ofvisual tasks. Specifically, we consider the pre-trained StyleGAN generator as alearned loss function and utilize its layer-wise disentangled representation totrain a novel hierarchical encoder. As a result, the visual feature produced byour encoder, termed as Generative Hierarchical Feature (GH-Feat), hascompelling discriminative and disentangled properties, facilitating a range ofboth discriminative and generative tasks. Extensive experiments on faceverification, landmark detection, layout prediction, transfer learning, stylemixing, and image editing show the appealing performance of the GH-Feat learnedfrom synthesizing images, outperforming existing unsupervised feature learningmethods.