PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

  • 2021-04-26 06:59:36
  • Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, Yonghong Tian
  • 16

Abstract

Large-scale Pretrained Language Models (PLMs) have become the new paradigmfor Natural Language Processing (NLP). PLMs with hundreds of billionsparameters such as GPT-3 have demonstrated strong performances on naturallanguage understanding and generation with \textit{few-shot in-context}learning. In this work, we present our practice on training large-scaleautoregressive language models named PanGu-$\alpha$, with up to 200 billionparameters. PanGu-$\alpha$ is developed under the MindSpore and trained on acluster of 2048 Ascend 910 AI processors. The training parallelism strategy isimplemented based on MindSpore Auto-parallel, which composes five parallelismdimensions to scale the training task to 2048 processors efficiently, includingdata parallelism, op-level model parallelism, pipeline model parallelism,optimizer model parallelism and rematerialization. To enhance thegeneralization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinesedata from a wide range of domains to pretrain the model. We empirically testthe generation ability of PanGu-$\alpha$ in various scenarios including textsummarization, question answering, dialogue generation, etc. Moreover, weinvestigate the effect of model scales on the few-shot performances across abroad range of Chinese NLP tasks. The experimental results demonstrate thesuperior capabilities of PanGu-$\alpha$ in performing various tasks underfew-shot or zero-shot settings.

 

Quick Read (beta)

loading the full paper ...