A Test for Evaluating Performance in Human-Computer Systems

  • 2022-06-24 18:44:58
  • Andres Campero, Michelle Vaccaro, Jaeyoon Song, Haoran Wen, Abdullah Almaatouq, Thomas W. Malone
  • 3

Abstract

The Turing test for comparing computer performance to that of humans is wellknown, but, surprisingly, there is no widely used test for comparing how muchbetter human-computer systems perform relative to humans alone, computersalone, or other baselines. Here, we show how to perform such a test using theratio of means as a measure of effect size. Then we demonstrate the use of thistest in three ways. First, in an analysis of 79 recently published experimentalresults, we find that, surprisingly, over half of the studies find a decreasein performance, the mean and median ratios of performance improvement are bothapproximately 1 (corresponding to no improvement at all), and the maximum ratiois 1.36 (a 36% improvement). Second, we experimentally investigate whether ahigher performance improvement ratio is obtained when 100 human programmersgenerate software using GPT-3, a massive, state-of-the-art AI system. In thiscase, we find a speed improvement ratio of 1.27 (a 27% improvement). Finally,we find that 50 human non-programmers using GPT-3 can perform the task about aswell as--and less expensively than--the human programmers. In this case,neither the non-programmers nor the computer would have been able to performthe task alone, so this is an example of a very strong form of human-computersynergy.

 

Quick Read (beta)

loading the full paper ...