Abstract
Diffusion models have achieved remarkable success in generatinghigh-resolution, realistic images across diverse natural distributions.However, their performance heavily relies on high-quality training data, makingit challenging to learn meaningful distributions from corrupted samples. Thislimitation restricts their applicability in scientific domains where clean datais scarce or costly to obtain. In this work, we introduce denoising scoredistillation (DSD), a surprisingly effective and novel approach for traininghigh-quality generative models from low-quality data. DSD first pretrains adiffusion model exclusively on noisy, corrupted samples and then distills itinto a one-step generator capable of producing refined, clean outputs. Whilescore distillation is traditionally viewed as a method to accelerate diffusionmodels, we show that it can also significantly enhance sample quality,particularly when starting from a degraded teacher model. Across varying noiselevels and datasets, DSD consistently improves generative performancewesummarize our empirical evidence in Fig. 1. Furthermore, we provide theoreticalinsights showing that, in a linear model setting, DSD identifies the eigenspaceof the clean data distributions covariance matrix, implicitly regularizing thegenerator. This perspective reframes score distillation as not only a tool forefficiency but also a mechanism for improving generative models, particularlyin low-quality data settings.