Emergent Abilities of Large Language Models

  • 2022-06-15 18:32:01
  • Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus
  • 115

Abstract

Scaling up language models has been shown to predictably improve performanceand sample efficiency on a wide range of downstream tasks. This paper insteaddiscusses an unpredictable phenomenon that we refer to as emergent abilities oflarge language models. We consider an ability to be emergent if it is notpresent in smaller models but is present in larger models. Thus, emergentabilities cannot be predicted simply by extrapolating the performance ofsmaller models. The existence of such emergence implies that additional scalingcould further expand the range of capabilities of language models.

 

Quick Read (beta)

loading the full paper ...