Abstract
Distributed learning is commonly used for training deep learning models,especially large models. In distributed learning, manual parallelism (MP)methods demand considerable human effort and have limited flexibility. Hence,automatic parallelism (AP) methods have recently been proposed for automatingthe parallel strategy optimization process. Existing AP methods suffer fromsub-optimal solutions because they do not jointly optimize the two categoriesof parallel strategies (i.e., inter-layer parallelism and intra-layerparallelism). In this paper, we propose a novel AP method called UniAP, whichunifies inter- and intra-layer automatic parallelism by mixed integer quadraticprogramming. To the best of our knowledge, UniAP is the first parallel methodthat can jointly optimize the two categories of parallel strategies to find anoptimal solution. Experimental results show that UniAP outperformsstate-of-the-art methods by up to 1.71$\times$ in throughput and reducesstrategy optimization time by up to 107$\times$ across five Transformer-basedmodels.