Unsupervised image-to-image translation is an inherently ill-posed problem.Recent methods based on deep encoder-decoder architectures have shownimpressive results, but we show that they only succeed due to a strong localitybias, and they fail to learn very simple nonlocal transformations (e.g. mappingupside down faces to upright faces). When the locality bias is removed, themethods are too powerful and may fail to learn simple local transformations. Inthis paper we introduce linear encoder-decoder architectures for unsupervisedimage to image translation. We show that learning is much easier and fasterwith these architectures and yet the results are surprisingly effective. Inparticular, we show a number of local problems for which the results of thelinear methods are comparable to those of state-of-the-art architectures butwith a fraction of the training time, and a number of nonlocal problems forwhich the state-of-the-art fails while linear methods succeed.