Abstract
Post-training of language models, either through reinforcement learning,preference optimization or supervised finetuning, tends to sharpen the outputprobability distribution and reduce the diversity of generated responses. Thisis particularly a problem for creative generative tasks where varied responsesare desired. In this work we introduce Diverse Preference Optimization (DivPO),an optimization method which learns to generate much more diverse responsesthan standard pipelines, while maintaining the quality of the generations. InDivPO, preference pairs are selected by first considering a pool of responses,and a measure of diversity among them, and selecting chosen examples as beingmore rare but high quality, while rejected examples are more common, but lowquality. DivPO results in generating 45.6% more diverse persona attributes, anda 74.6% increase in story diversity, while maintaining similar win rates asstandard baselines. On general instruction following, DivPO results in a 46.2%increase in diversity, and a 2.4% winrate improvement compared to DPO.