Uncertainty quantification (UQ) is an important component of molecularproperty prediction, particularly for drug discovery applications where modelpredictions direct experimental design and where unanticipated imprecisionwastes valuable time and resources. The need for UQ is especially acute forneural models, which are becoming increasingly standard yet are challenging tointerpret. While several approaches to UQ have been proposed in the literature,there is no clear consensus on the comparative performance of these models. Inthis paper, we study this question in the context of regression tasks. Wesystematically evaluate several methods on five benchmark datasets usingmultiple complementary performance metrics. Our experiments show that none ofthe methods we tested is unequivocally superior to all others, and noneproduces a particularly reliable ranking of errors across multiple datasets.While we believe these results show that existing UQ methods are not sufficientfor all common use-cases and demonstrate the benefits of further research, weconclude with a practical recommendation as to which existing techniques seemto perform well relative to others.