Abstract
Reinforcement Learning (RL) is highly dependent on the meticulous design ofthe reward function. However, accurately assigning rewards to each state-actionpair in Long-Term RL (LTRL) challenges is formidable. Consequently, RL agentsare predominantly trained with expert guidance. Drawing on the principles ofordinal utility theory from economics, we propose a novel reward estimationalgorithm: ELO-Rating based RL (ERRL). This approach is distinguished by twomain features. Firstly, it leverages expert preferences over trajectoriesinstead of cardinal rewards (utilities) to compute the ELO rating of eachtrajectory as its reward. Secondly, a new reward redistribution algorithm isintroduced to mitigate training volatility in the absence of a fixed anchorreward. Our method demonstrates superior performance over several leadingbaselines in long-term scenarios (extending up to 5000 steps), whereconventional RL algorithms falter. Furthermore, we conduct a thorough analysisof how expert preferences affect the outcomes.