A Systematic Evaluation of Large Language Models of Code

  • 2022-05-04 17:08:31
  • Frank F. Xu, Uri Alon, Graham Neubig, Vincent J. Hellendoorn
  • 0

Abstract

Large language models (LMs) of code have recently shown tremendous promise incompleting code and synthesizing code from natural language descriptions.However, the current state-of-the-art code LMs (e.g., Codex (Chen et al.,2021)) are not publicly available, leaving many questions about their model anddata design decisions. We aim to fill in some of these blanks through asystematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo,GPT-NeoX-20B, and CodeParrot, across various programming languages. AlthoughCodex itself is not open-source, we find that existing open-source models doachieve close results in some programming languages, although targeted mainlyfor natural language modeling. We further identify an important missing piecein the form of a large open-source model trained exclusively on a multi-lingualcorpus of code. We release a new model, PolyCoder, with 2.7B parameters basedon the GPT-2 architecture, which was trained on 249GB of code across 12programming languages on a single machine. In the C programming language,PolyCoder outperforms all models including Codex. Our trained models areopen-source and publicly available at https://github.com/VHellendoorn/Code-LMs,which enables future research and application in this area.

 

Quick Read (beta)

loading the full paper ...