Abstract
Reinforcement Learning from Human Feedback (RLHF) has recently surged inpopularity, particularly for aligning large language models and other AIsystems with human intentions. At its core, RLHF can be viewed as a specializedinstance of Preference-based Reinforcement Learning (PbRL), where thepreferences specifically originate from human judgments rather than arbitraryevaluators. Despite this connection, most existing approaches in both RLHF andPbRL primarily focus on optimizing a mean reward objective, neglectingscenarios that necessitate risk-awareness, such as AI safety, healthcare, andautonomous driving. These scenarios often operate under a one-episode-rewardsetting, which makes conventional risk-sensitive objectives inapplicable. Toaddress this, we explore and prove the applicability of two risk-awareobjectives to PbRL : nested and static quantile risk objectives. We alsointroduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize bothnested and static objectives. Additionally, we provide a theoretical analysisof the regret upper bounds, demonstrating that they are sublinear with respectto the number of episodes, and present empirical results to support ourfindings. Our code is available inhttps://github.com/aguilarjose11/PbRLNeurips.