Abstract
Video foundation models have achieved significant advancement with the helpof network upgrade as well as model scale-up. However, they are still hard tomeet requirements of applications due to unsatisfied generation quality. Tosolve this problem, we propose to align video foundation models with humanpreferences from the perspective of post-training in this paper. Consequently,we introduce an Iterative Preference Optimization strategy to enhance generatedvideo quality by incorporating human feedback. Specifically, IPO exploits acritic model to justify video generations for pairwise ranking as in DirectPreference Optimization or point-wise scoring as in Kahneman-TverskyOptimization. Given this, IPO optimizes video foundation models with guidanceof signals from preference feedback, which helps improve generated videoquality in subject consistency, motion smoothness and aesthetic quality, etc.In addition, IPO incorporates the critic model with the multi-modality largelanguage model, which enables it to automatically assign preference labelswithout need of retraining or relabeling. In this way, IPO can efficientlyperform multi-round preference optimization in an iterative manner, without theneed of tediously manual labeling. Comprehensive experiments demonstrate thatthe proposed IPO can effectively improve the video generation quality of apretrained model and help a model with only 2B parameters surpass the one with5B parameters. Besides, IPO achieves new state-of-the-art performance on VBenchbenchmark. We will release our source codes, models as well as dataset toadvance future research and applications.