Abstract
In recent years, the use of deep learning in language models, textauto-completion, and text generation has made tremendous progress and gainedmuch attention from the research community. Some products and research projectsclaim that they can generate text that can be interpreted as human-writing,enabling new possibilities in many application areas. Among the different areasrelated to language processing, one of the most notable in applying this typeof modeling is the processing of programming languages. For years, the MachineLearning community has been researching in this Big Code area, pursuing goalslike applying different approaches to auto-complete generate, fix, or evaluatecode programmed by humans. One of the approaches followed in recent years topursue these goals is the use of Deep-Learning-enabled language models.Considering the increasing popularity of that approach, we detected a lack ofempirical papers that compare different methods and deep learning architecturesto create and use language models based on programming code. In this paper, wecompare different neural network (NN) architectures like AWD-LSTMs, AWD-QRNNs,and Transformer, while using transfer learning, and different tokenizations tosee how they behave in building language models using a Python dataset for codegeneration and filling mask tasks. Considering the results, we discuss thedifferent strengths and weaknesses of each approach and technique and whatlacks do we find to evaluate the language models or apply them in a realprogramming context while including humans-in-the-loop.