Abstract
We present Natural Gradient Boosting (NGBoost), an algorithm which bringsprobabilistic prediction capability to gradient boosting in a generic way.Predictive uncertainty estimation is crucial in many applications such ashealthcare and weather forecasting. Probabilistic prediction, which is theapproach where the model outputs a full probability distribution over theentire outcome space, is a natural way to quantify those uncertainties.Gradient Boosting Machines have been widely successful in prediction tasks onstructured input data, but a simple boosting solution for probabilisticprediction of real valued outputs is yet to be made. NGBoost is a gradientboosting approach which uses the \emph{Natural Gradient} to address technicalchallenges that makes generic probabilistic prediction hard with existinggradient boosting methods. Our approach is modular with respect to the choiceof base learner, probability distribution, and scoring rule. We showempirically on several regression datasets that NGBoost provides competitivepredictive performance of both uncertainty estimates and traditional metrics.