Abstract
Diffusion models have enabled remarkably high-quality medical imagegeneration, yet it is challenging to enforce anatomical constraints ingenerated images. This hampers many useful applications, includingpre-registered image generation, counterfactual scenarios, and others. To thisend, we propose a diffusion model-based method that supportsanatomically-controllable medical image generation, by following a multi-classanatomical segmentation mask at each sampling step. We additionally introduce arandom mask ablation training algorithm to enable conditioning on a selectedcombination of anatomical constraints while allowing flexibility in otheranatomical areas. We compare our model ("Seg-Diff") to existing methods onbreast MRI and abdominal/neck-to-pelvis CT datasets with a wide range ofanatomical objects. Results show that it reaches a new state-of-the-art in thefaithfulness of generated images to input anatomical masks on both datasets,and is on par for general anatomical realism. Finally, our model also enjoysthe extra benefit of being able to adjust the anatomical similarity ofgenerated images to real images of choice through interpolation in its latentspace.