Abstract
Random Forests (RF) are among the state-of-the-art in many machine learningapplications. With the ongoing integration of ML models into everyday life, thedeployment and continuous application of models becomes more and more animportant issue. Hence, small models which offer good predictive performancebut use small amounts of memory are required. Ensemble pruning is a standardtechnique to remove unnecessary classifiers from an ensemble to reduce theoverall resource consumption and sometimes even improve the performance of theoriginal ensemble. In this paper, we revisit ensemble pruning in the context of`modernly' trained Random Forests where trees are very large. We show that theimprovement effects of pruning diminishes for ensembles of large trees but thatpruning has an overall better accuracy-memory trade-off than RF. However,pruning does not offer fine-grained control over this trade-off because itremoves entire trees from the ensemble. To further improve the accuracy-memorytrade-off we present a simple, yet surprisingly effective algorithm thatrefines the predictions in the leaf nodes in the forest via stochastic gradientdescent. We evaluate our method against 7 state-of-the-art pruning methods andshow that our method outperforms the other methods on 11 of 16 datasets with astatistically significant better accuracy-memory trade-off compared to mostmethods. We conclude our experimental evaluation with a case study showing thatour method can be applied in a real-world setting.