Abstract
Conventional score-based diffusion models (DMs) may struggle with anisotropicGaussian diffusion processes due to the required inversion of covariancematrices in the denoising score matching training objective\cite{vincent_connection_2011}. We propose Whitened Score (WS) diffusionmodels, a novel SDE-based framework that learns the Whitened Score functioninstead of the standard score. This approach circumvents covariance inversion,extending score-based DMs by enabling stable training of DMs on arbitraryGaussian forward noising processes. WS DMs establish equivalence with FM forarbitrary Gaussian noise, allow for tailored spectral inductive biases, andprovide strong Bayesian priors for imaging inverse problems with structurednoise. We experiment with a variety of computational imaging tasks using theCIFAR and CelebA ($64\times64$) datasets and demonstrate that WS diffusionpriors trained on anisotropic Gaussian noising processes consistentlyoutperform conventional diffusion priors based on isotropic Gaussian noise.