Abstract
In the era of Large Language Models (LLMs), alignment has emerged as afundamental yet challenging problem in the pursuit of more reliable,controllable, and capable machine intelligence. The recent success of reasoningmodels and conversational AI systems has underscored the critical role ofreinforcement learning (RL) in enhancing these systems, driving increasedresearch interest at the intersection of RL and LLM alignment. This paperprovides a comprehensive review of recent advances in LLM alignment through thelens of inverse reinforcement learning (IRL), emphasizing the distinctionsbetween RL techniques employed in LLM alignment and those in conventional RLtasks. In particular, we highlight the necessity of constructing neural rewardmodels from human data and discuss the formal and practical implications ofthis paradigm shift. We begin by introducing fundamental concepts in RL toprovide a foundation for readers unfamiliar with the field. We then examinerecent advances in this research agenda, discussing key challenges andopportunities in conducting IRL for LLM alignment. Beyond methodologicalconsiderations, we explore practical aspects, including datasets, benchmarks,evaluation metrics, infrastructure, and computationally efficient training andinference techniques. Finally, we draw insights from the literature onsparse-reward RL to identify open questions and potential research directions.By synthesizing findings from diverse studies, we aim to provide a structuredand critical overview of the field, highlight unresolved challenges, andoutline promising future directions for improving LLM alignment through RL andIRL techniques.