Numerous style transfer methods which produce artistic styles of portraitshave been proposed to date. However, the inverse problem of converting thestylized portraits back into realistic faces is yet to be investigatedthoroughly. Reverting an artistic portrait to its original photo-realistic faceimage has potential to facilitate human perception and identity analysis. Inthis paper, we propose a novel Face Destylization Neural Network (FDNN) torestore the latent photo-realistic faces from the stylized ones. We develop aStyle Removal Network composed of convolutional, fully-connected anddeconvolutional layers. The convolutional layers are designed to extract facialcomponents from stylized face images. Consecutively, the fully-connected layertransfers the extracted feature maps of stylized images into the correspondingfeature maps of real faces and the deconvolutional layers generate real facesfrom the transferred feature maps. To enforce the destylized faces to besimilar to authentic face images, we employ a discriminative network, whichconsists of convolutional and fully connected layers. We demonstrate theeffectiveness of our network by conducting experiments on an extensive set ofsynthetic images. Furthermore, we illustrate our network can recover faces fromstylized portraits and real paintings for which the stylized data wasunavailable during the training phase.