Abstract
Reinforcement Learning from Human Feedback (RLHF) has shown promise inaligning large language models (LLMs). Yet its reliance on a singular rewardmodel often overlooks the diversity of human preferences. Recent approachesaddress this limitation by leveraging multi-dimensional feedback to fine-tunecorresponding reward models and train LLMs using reinforcement learning.However, the process is costly and unstable, especially given the competing andheterogeneous nature of human preferences. In this paper, we propose MixingPreference Optimization (MPO), a post-processing framework for aggregatingsingle-objective policies as an alternative to both multi-objective RLHF(MORLHF) and MaxMin-RLHF. MPO avoids alignment from scratch. Instead, itlog-linearly combines existing policies into a unified one with the weight ofeach policy computed via a batch stochastic mirror descent. Empirical resultsdemonstrate that MPO achieves balanced performance across diverse preferences,outperforming or matching existing models with significantly reducedcomputational costs.