Gradient boosted decision trees are some of the most popular algorithms inapplied machine learning. They are a flexible and powerful tool that canrobustly fit to any tabular dataset in a scalable and computationally efficientway. One of the most critical parameters to tune when fitting these models arethe various penalty terms used to distinguish signal from noise in the currentmodel. These penalties are effective in practice, but are lacking in robusttheoretical justifications. In this paper we develop and present a noveltheoretically justified hypothesis test of split quality for gradient boostedtree ensembles and demonstrate that using this method instead of the commonpenalty terms leads to a significant reduction in out of sample loss.Additionally, this method provides a theoretically well-justified stoppingcondition for the tree growing algorithm. We also present several innovativeextensions to the method, opening the door for a wide variety of novel treepruning algorithms.