Abstract
Reinforcement Learning from human feedback (RLHF) has been shown a promisingdirection for aligning generative models with human intent and has also beenexplored in recent works for alignment of diffusion generative models. In thiswork, we provide a rigorous treatment by formulating the task of fine-tuningdiffusion models, with reward functions learned from human feedback, as anexploratory continuous-time stochastic control problem. Our key idea lies intreating the score-matching functions as controls/actions, and upon this, wedevelop a unified framework from a continuous-time perspective, to employreinforcement learning (RL) algorithms in terms of improving the generationquality of diffusion models. We also develop the corresponding continuous-timeRL theory for policy optimization and regularization under assumptions ofstochastic different equations driven environment. Experiments on thetext-to-image (T2I) generation will be reported in the accompanied paper.