Abstract
Large Language Models (LLMs) exhibit remarkably powerful capabilities. One ofthe crucial factors to achieve success is aligning the LLM's output with humanpreferences. This alignment process often requires only a small amount of datato efficiently enhance the LLM's performance. While effective, research in thisarea spans multiple domains, and the methods involved are relatively complex tounderstand. The relationships between different methods have beenunder-explored, limiting the development of the preference alignment. In lightof this, we break down the existing popular alignment strategies into differentcomponents and provide a unified framework to study the current alignmentstrategies, thereby establishing connections among them. In this survey, wedecompose all the strategies in preference learning into four components:model, data, feedback, and algorithm. This unified view offers an in-depthunderstanding of existing alignment algorithms and also opens up possibilitiesto synergize the strengths of different strategies. Furthermore, we presentdetailed working examples of prevalent existing algorithms to facilitate acomprehensive understanding for the readers. Finally, based on our unifiedperspective, we explore the challenges and future research directions foraligning large language models with human preferences.