Abstract

Scaling up language models has been shown to predictably improve performanceand sample efficiency on a wide range of downstream tasks. This paper insteaddiscusses an unpredictable phenomenon that we refer to as emergent abilities oflarge language models. We consider an ability to be emergent if it is notpresent in smaller models but is present in larger models. Thus, emergentabilities cannot be predicted simply by extrapolating the performance ofsmaller models. The existence of such emergence implies that additional scalingcould further expand the range of capabilities of language models.

 

Quick Read (beta)