Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

  • 2019-04-12 17:18:08
  • Isidro Cortes-Ciriano, Andreas Bender
  • 3


While the use of deep learning in drug discovery is gaining increasingattention, the lack of methods to compute reliable errors in prediction forNeural Networks prevents their application to guide decision making in domainswhere identifying unreliable predictions is essential, e.g. precision medicine.Here, we present a framework to compute reliable errors in prediction forNeural Networks using Test-Time Dropout and Conformal Prediction. Specifically,the algorithm consists of training a single Neural Network using dropout, andthen applying it N times to both the validation and test sets, also employingdropout in this step. Therefore, for each instance in the validation and testsets an ensemble of predictions were generated. The residuals and absoluteerrors in prediction for the validation set were then used to computeprediction errors for test set instances using Conformal Prediction. We showusing 24 bioactivity data sets from ChEMBL 23 that dropout Conformal Predictorsare valid (i.e., the fraction of instances whose true value lies within thepredicted interval strongly correlates with the confidence level) andefficient, as the predicted confidence intervals span a narrower set of valuesthan those computed with Conformal Predictors generated using Random Forest(RF) models. Lastly, we show in retrospective virtual screening experimentsthat dropout and RF-based Conformal Predictors lead to comparable retrievalrates of active compounds. Overall, we propose a computationally efficientframework (as only N extra forward passes are required in addition to traininga single network) to harness Test-Time Dropout and the Conformal Predictionframework, and to thereby generate reliable prediction errors for deep NeuralNetworks.


Quick Read (beta)

loading the full paper ...