Abstract
This paper explores the limits of the current generation of large languagemodels for program synthesis in general purpose programming languages. Weevaluate a collection of such models (with between 244M and 137B parameters) ontwo new benchmarks, MBPP and MathQA-Python, in both the few-shot andfine-tuning regimes. Our benchmarks are designed to measure the ability ofthese models to synthesize short Python programs from natural languagedescriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974programming tasks, designed to be solvable by entry-level programmers. TheMathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914problems that evaluate the ability of the models to synthesize code from morecomplex text. On both datasets, we find that synthesis performance scaleslog-linearly with model size. Our largest models, even without finetuning on acode dataset, can synthesize solutions to 59.6 percent of the problems fromMBPP using few-shot learning with a well-designed prompt. Fine-tuning on aheld-out portion of the dataset improves performance by about 10 percentagepoints across most model sizes. On the MathQA-Python dataset, the largestfine-tuned model achieves 83.8 percent accuracy. Going further, we study themodel's ability to engage in dialog about code, incorporating human feedback toimprove its solutions. We find that natural language feedback from a humanhalves the error rate compared to the model's initial prediction. Additionally,we conduct an error analysis to shed light on where these models fall short andwhat types of programs are most difficult to generate. Finally, we explore thesemantic grounding of these models by fine-tuning them to predict the resultsof program execution. We find that even our best models are generally unable topredict the output of a program given a specific input.