Abstract
Recent advances in modern Natural Language Processing (NLP) research havebeen dominated by the combination of Transfer Learning methods with large-scaleTransformer language models. With them came a paradigm shift in NLP with thestarting point for training a model on a downstream task moving from a blankspecific model to a general-purpose pretrained architecture. Still, creatingthese general-purpose models remains an expensive and time-consuming processrestricting the use of these methods to a small sub-set of the wider NLPcommunity. In this paper, we present Transformers, a library forstate-of-the-art NLP, making these developments available to the community bygathering state-of-the-art general-purpose pretrained models under a unifiedAPI together with an ecosystem of libraries, examples, tutorials and scriptstargeting many downstream NLP tasks. Transformers features carefully craftedmodel implementations and high-performance pretrained weights for two main deeplearning frameworks, PyTorch and TensorFlow, while supporting all the necessarytools to analyze, evaluate and use these models in downstream tasks such astext/token classification, questions answering and language generation amongothers. Transformers has gained significant organic traction and adoption amongboth the researcher and practitioner communities. We are committed at HuggingFace to pursue the efforts to develop Transformers with the ambition ofcreating the standard library for building NLP systems.