We consider the problem of segmenting a biomedical image into anatomicalregions of interest. We specifically address the frequent scenario where wehave no paired training data that contains images and their manualsegmentations. Instead, we employ unpaired segmentation images to build ananatomical prior. Critically these segmentations can be derived from imagingdata from a different dataset and imaging modality than the current task. Weintroduce a generative probabilistic model that employs the learned priorthrough a convolutional neural network to compute segmentations in anunsupervised setting. We conducted an empirical analysis of the proposedapproach in the context of structural brain MRI segmentation, using amulti-study dataset of more than 14,000 scans. Our results show that ananatomical prior can enable fast unsupervised segmentation which is typicallynot possible using standard convolutional networks. The integration ofanatomical priors can facilitate CNN-based anatomical segmentation in a rangeof novel clinical problems, where few or no annotations are available and thusstandard networks are not trainable. The code is freely available athttp://github.com/adalca/neuron.