Abstract
In recommender systems such as news feed stream, it is essential to optimizethe long-term utilities in the continuous user-system interaction processes.Previous works have proved the capability of reinforcement learning in thisproblem. However, there are many practical challenges to implement deepreinforcement learning in online systems, including low sample efficiency,uncontrollable risks, and excessive variances. To address these issues, wepropose a novel reinforcement learning method, namely model-basedcounterfactual advantage learning (MBCAL). The proposed method takes advantageof the characteristics of recommender systems and draws ideas from themodel-based reinforcement learning method for higher sample efficiency. It hastwo components: an environment model that predicts the instant user behaviorone-by-one in an auto-regressive form, and a future advantage model thatpredicts the future utility. To alleviate the impact of excessive variance whenlearning the future advantage model, we employ counterfactual comparisonsderived from the environment model. In consequence, the proposed methodpossesses high sample efficiency and significantly lower variance; Also, it isable to use existing user logs to avoid the risks of starting from scratch. Incontrast to its capability, its implementation cost is relatively low, whichfits well with practical systems. Theoretical analysis and elaborateexperiments are presented. Results show that the proposed method transcends theother supervised learning and RL-based methods in both sample efficiency andasymptotic performances.