Abstract
In this work, we present a new network design paradigm. Our goal is to helpadvance the understanding of network design and discover design principles thatgeneralize across settings. Instead of focusing on designing individual networkinstances, we design network design spaces that parametrize populations ofnetworks. The overall process is analogous to classic manual design ofnetworks, but elevated to the design space level. Using our methodology weexplore the structure aspect of network design and arrive at a low-dimensionaldesign space consisting of simple, regular networks that we call RegNet. Thecore insight of the RegNet parametrization is surprisingly simple: widths anddepths of good networks can be explained by a quantized linear function. Weanalyze the RegNet design space and arrive at interesting findings that do notmatch the current practice of network design. The RegNet design space providessimple and fast networks that work well across a wide range of flop regimes.Under comparable training settings and flops, the RegNet models outperform thepopular EfficientNet models while being up to 5x faster on GPUs.