UER: An Open-Source Toolkit for Pre-training Models

  • 2019-09-12 13:46:58
  • Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du
  • 17

Abstract

Existing works, including ELMO and BERT, have revealed the importance ofpre-training for NLP tasks. While there does not exist a single pre-trainingmodel that works best in all cases, it is of necessity to develop a frameworkthat is able to deploy various pre-training models efficiently. For thispurpose, we propose an assemble-on-demand pre-training toolkit, namelyUniversal Encoder Representations (UER). UER is loosely coupled, andencapsulated with rich modules. By assembling modules on demand, users caneither reproduce a state-of-the-art pre-training model or develop apre-training model that remains unexplored. With UER, we have built a modelzoo, which contains pre-trained models based on different corpora, encoders,and targets (objectives). With proper pre-trained models, we could achieve newstate-of-the-art results on a range of downstream datasets.

 

Quick Read (beta)

loading the full paper ...