Abstract
The generation of realistic medical images from text descriptions hassignificant potential to address data scarcity challenges in healthcare AIwhile preserving patient privacy. This paper presents a comprehensive study oftext-to-image synthesis in the medical domain, comparing two distinctapproaches: (1) fine-tuning large pre-trained latent diffusion models and (2)training small, domain-specific models. We introduce a novel model named MSDM,an optimized architecture based on Stable Diffusion that integrates a clinicaltext encoder, variational autoencoder, and cross-attention mechanisms to betteralign medical text prompts with generated images. Our study compares twoapproaches: fine-tuning large pre-trained models (FLUX, Kandinsky) versustraining compact domain-specific models (MSDM). Evaluation across colonoscopy(MedVQA-GI) and radiology (ROCOv2) datasets reveals that while large modelsachieve higher fidelity, our optimized MSDM delivers comparable quality withlower computational costs. Quantitative metrics and qualitative evaluations bymedical experts reveal strengths and limitations of each approach.