Abstract
In this paper I re-examine the COMPAS recidivism score and criminal historydata collected by ProPublica in 2016, which has fueled intense debate andresearch in the nascent field of `algorithmic fairness' or `fair machinelearning' over the past three years. ProPublica's COMPAS data is used in anever-increasing number of studies to test various definitions and methodologiesof algorithmic fairness. This paper takes a closer look at the actual datasetsput together by ProPublica. By doing so, I find that ProPublica made animportant data processing mistake when it created some of the key datasets mostoften used by other researchers. In particular, the datasets built to study thelikelihood of recidivism within two years of the original COMPAS screeningdate. As I show in this paper, ProPublica made a mistake implementing thetwo-year sample cutoff rule for recidivists in such datasets (whereas itimplemented an appropriate two-year sample cutoff rule for non-recidivists). Asa result, ProPublica incorrectly kept a disproportionate share of recidivists.This data processing mistake leads to biased two-year recidivism datasets, withartificially high recidivism rates. This also affects the positive and negativepredictive values. On the other hand, this data processing mistake does notimpact some of the key statistical measures highlighted by ProPublica and otherresearchers, such as the false positive and false negative rates, nor theoverall accuracy.