Abstract
Deep Learning (DL) models are becoming larger, because the increase in modelsize might offer significant accuracy gain. To enable the training of largedeep networks, data parallelism and model parallelism are two well-knownapproaches for parallel training. However, data parallelism does not helpreduce memory footprint per device. In this work, we introduce Large deep 3DConvNets with Automated Model Parallelism (LAMP) and investigate the impact ofboth input's and deep 3D ConvNets' size on segmentation accuracy. Throughautomated model parallelism, it is feasible to train large deep 3D ConvNetswith a large input patch, even the whole image. Extensive experimentsdemonstrate that, facilitated by the automated model parallelism, thesegmentation accuracy can be improved through increasing model size and inputcontext size, and large input yields significant inference speedup comparedwith sliding window of small patches in the inference. Code isavailable\footnote{https://monai.io/research/lamp-automated-model-parallelism}.