Artificial intelligence (AI) has been successfully applied in numerousscientific domains including biomedicine and healthcare. Here, it has led toseveral breakthroughs ranging from clinical decision support systems, imageanalysis to whole genome sequencing. However, training an AI model on sensitivedata raises also concerns about the privacy of individual participants.Adversary AIs, for example, can abuse even summary statistics of a study todetermine the presence or absence of an individual in a given dataset. This hasresulted in increasing restrictions to access biomedical data, which in turn isdetrimental for collaborative research and impedes scientific progress. Hencethere has been an explosive growth in efforts to harness the power of AI forlearning from sensitive data while protecting patients' privacy. This paperprovides a structured overview of recent advances in privacy-preserving AItechniques in biomedicine. It places the most important state-of-the-artapproaches within a unified taxonomy, and discusses their strengths,limitations, and open problems.