Abstract
In reinforcement learning (RL), agents continually interact with theenvironment and use the feedback to refine their behavior. To guide policyoptimization, reward models are introduced as proxies of the desiredobjectives, such that when the agent maximizes the accumulated reward, it alsofulfills the task designer's intentions. Recently, significant attention fromboth academic and industrial researchers has focused on developing rewardmodels that not only align closely with the true objectives but also facilitatepolicy optimization. In this survey, we provide a comprehensive review ofreward modeling techniques within the deep RL literature. We begin by outliningthe background and preliminaries in reward modeling. Next, we present anoverview of recent reward modeling approaches, categorizing them based on thesource, the mechanism, and the learning paradigm. Building on thisunderstanding, we discuss various applications of these reward modelingtechniques and review methods for evaluating reward models. Finally, weconclude by highlighting promising research directions in reward modeling.Altogether, this survey includes both established and emerging methods, fillingthe vacancy of a systematic review of reward models in current literature.