Abstract
Calibration is defined as the ratio of the average predicted click rate tothe true click rate. The optimization of calibration is essential to manyonline advertising recommendation systems because it directly affects thedownstream bids in ads auctions and the amount of money charged to advertisers.Despite its importance, calibration optimization often suffers from a problemcalled "maximization bias". Maximization bias refers to the phenomenon that themaximum of predicted values overestimates the true maximum. The problem isintroduced because the calibration is computed on the set selected by theprediction model itself. It persists even if unbiased predictions can beachieved on every datapoint and worsens when covariate shifts exist between thetraining and test sets. To mitigate this problem, we theorize thequantification of maximization bias and propose a variance-adjusting debiasing(VAD) meta-algorithm in this paper. The algorithm is efficient, robust, andpractical as it is able to mitigate maximization bias problems under covariateshifts, neither incurring additional online serving costs nor compromising theranking performance. We demonstrate the effectiveness of the proposed algorithmusing a state-of-the-art recommendation neural network model on a large-scalereal-world dataset.