Abstract
The ranking utility function in an ad recommender system, which linearlycombines predictions of various business goals, plays a central role inbalancing values across the platform, advertisers, and users. Traditionalmanual tuning, while offering simplicity and interpretability, often yieldssuboptimal results due to its unprincipled tuning objectives, the vast amountof parameter combinations, and its lack of personalization and adaptability toseasonality. In this work, we propose a general Deep Reinforcement Learningframework for Personalized Utility Tuning (DRL-PUT) to address the challengesof multi-objective optimization within ad recommender systems. Our keycontributions include: 1) Formulating the problem as a reinforcement learningtask: given the state of an ad request, we predict the optimal hyperparametersto maximize a pre-defined reward. 2) Developing an approach to directly learnan optimal policy model using online serving logs, avoiding the need toestimate a value function, which is inherently challenging due to the highvariance and unbalanced distribution of immediate rewards. We evaluated DRL-PUTthrough an online A/B experiment in Pinterest's ad recommender system. Comparedto the baseline manual utility tuning approach, DRL-PUT improved theclick-through rate by 9.7% and the long click-through rate by 7.7% on thetreated segment. We conducted a detailed ablation study on the impact ofdifferent reward definitions and analyzed the personalization aspect of thelearned policy model.