Abstract
Recent advances in modern Natural Language Processing (NLP) research havebeen dominated by the combination of Transfer Learning methods with large-scalelanguage models, in particular based on the Transformer architecture. With themcame a paradigm shift in NLP with the starting point for training a model on adownstream task moving from a blank specific model to a general-purposepretrained architecture. Still, creating these general-purpose models remainsan expensive and time-consuming process restricting the use of these methods toa small sub-set of the wider NLP community. In this paper, we presentHuggingFace's Transformers library, a library for state-of-the-art NLP, makingthese developments available to the community by gathering state-of-the-artgeneral-purpose pretrained models under a unified API together with anecosystem of libraries, examples, tutorials and scripts targeting manydownstream NLP tasks. HuggingFace's Transformers library features carefullycrafted model implementations and high-performance pretrained weights for twomain deep learning frameworks, PyTorch and TensorFlow, while supporting all thenecessary tools to analyze, evaluate and use these models in downstream taskssuch as text/token classification, questions answering and language generationamong others. The library has gained significant organic traction and adoptionamong both the researcher and practitioner communities. We are committed atHuggingFace to pursue the efforts to develop this toolkit with the ambition ofcreating the standard library for building NLP systems. HuggingFace'sTransformers library is available at\url{https://github.com/huggingface/transformers}.