Abstract
Double machine learning provides $\sqrt{n}$-consistent estimates ofparameters of interest even when high-dimensional or nonparametric nuisanceparameters are estimated at an $n^{-1/4}$ rate. The key is to employNeyman-orthogonal moment equations which are first-order insensitive toperturbations in the nuisance parameters. We show that the $n^{-1/4}$requirement can be improved to $n^{-1/(2k+2)}$ by employing a $k$-th ordernotion of orthogonality that grants robustness to more complex orhigher-dimensional nuisance parameters. In the partially linear regressionsetting popular in causal inference, we show that we can construct second-orderorthogonal moments if and only if the treatment residual is not normallydistributed. Our proof relies on Stein's lemma and may be of independentinterest. We conclude by demonstrating the robustness benefits of an explicitdoubly-orthogonal estimation procedure for treatment effect.