Measuring Massive Multitask Language Understanding

  • 2020-09-07 17:59:25
  • Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt
  • 41


We propose a new test to measure a text model's multitask accuracy. The testcovers 57 tasks including elementary mathematics, US history, computer science,law, and more. To attain high accuracy on this test, models must possessextensive world knowledge and problem solving ability. We find that while mostrecent models have near random-chance accuracy, the very largest GPT-3 modelimproves over random chance by almost 20 percentage points on average. However,on every one of the 57 tasks, the best models still need substantialimprovements before they can reach human-level accuracy. Models also havelopsided performance and frequently do not know when they are wrong. Worse,they still have near-random accuracy on some socially important subjects suchas morality and law. By comprehensively evaluating the breadth and depth of amodel's academic and professional understanding, our test can be used toanalyze models across many tasks and to identify important shortcomings.


Quick Read (beta)

loading the full paper ...