Abstract
We consider approximation rates of sparsely connected deep rectified linearunit (ReLU) and rectified power unit (RePU) neural networks for functions inBesov spaces $B^\alpha_{q}(L^p)$ in arbitrary dimension $d$, on bounded orunbounded domains. We show that RePU networks with a fixed activation functionattain optimal approximation rates for functions in the Besov space$B^\alpha_{\tau}(L^\tau)$ on the critical embedding line $1/\tau=\alpha/d+1/p$for arbitrary smoothness order $\alpha>0$. Moreover, we show that ReLU networksattain near to optimal rates for any Besov space strictly above the criticalline. Using interpolation theory, this implies that the entire range ofsmoothness classes at or above the critical line is (near to) optimallyapproximated by deep ReLU/RePU networks.