Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising

  • 2018-08-07 05:15:08
  • Di Wu, Xiujun Chen, Xun Yang, Hao Wang, Qing Tan, Xiaoxun Zhang, Jian Xu, Kun Gai
  • 0

Abstract

Real-time bidding (RTB) is an important mechanism in online displayadvertising, where a proper bid for each page view plays an essential role forgood marketing results. Budget constrained bidding is a typical scenario in RTBwhere the advertisers hope to maximize the total value of the winningimpressions under a pre-set budget constraint. However, the optimal biddingstrategy is hard to be derived due to the complexity and volatility of theauction environment. To address these challenges, in this paper, we formulatebudget constrained bidding as a Markov Decision Process and propose amodel-free reinforcement learning framework to resolve the optimizationproblem. Our analysis shows that the immediate reward from environment ismisleading under a critical resource constraint. Therefore, we innovate areward function design methodology for the reinforcement learning problems withconstraints. Based on the new reward design, we employ a deep neural network tolearn the appropriate reward so that the optimal policy can be learnedeffectively. Different from the prior model-based work, which suffers from thescalability problem, our framework is easy to be deployed in large-scaleindustrial applications. The experimental evaluations demonstrate theeffectiveness of our framework on large-scale real datasets.

 

Quick Read (beta)

loading the full paper ...