Style transfer usually refers to the task of applying color and textureinformation from a specific style image to a given content image whilepreserving the structure of the latter. Here we tackle the more generic problemof semantic style transfer: given two unpaired collections of images, we aim tolearn a mapping between the corpus-level style of each collection, whilepreserving semantic content shared across the two domains. We introduce XGAN("Cross-GAN"), a dual adversarial autoencoder, which captures a sharedrepresentation of the common domain semantic content in an unsupervised way,while jointly learning the domain-to-domain image translations in bothdirections. We exploit ideas from the domain adaptation literature and define asemantic consistency loss which encourages the model to preserve semantics inthe learned embedding space. We report promising qualitative results for thetask of face-to-cartoon translation. The cartoon dataset, CartoonSet, wecollected for this purpose is publicly available atgoogle.github.io/cartoonset/ as a new benchmark for semantic style transfer.