Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs

  • 2018-03-12 16:04:28
  • Mike Voets, Kajsa Møllersen, Lars Ailo Bongo
  • 4


We have replicated some experiments in 'Development and validation of a deeplearning algorithm for detection of diabetic retinopathy in retinal fundusphotographs' that was published in JAMA 2016; 316(22). We re-implemented themethods since the source code is not available. The original study used fundus images from EyePACS and three hospitals inIndia for training their detection algorithm. We used a different EyePACS dataset that was made available in a Kaggle competition. For evaluating thealgorithm's performance the benchmark data set Messidor-2 was used. We used thesimilar Messidor-Original data set to evaluate our algorithm's performance. Inthe original study licensed ophthalmologists re-graded all their obtainedimages for diabetic retinopathy, macular edema, and image gradability. Ourchallenge was to re-implement the methods with publicly available data sets andone diabetic retinopathy grade per image, find the hyper-parameter settings fortraining and validation that were not described in the original study, and makean assessment on the impact of training with ungradable images. We were not able to reproduce the performance as reported in the originalstudy. We believe our model did not learn to recognize lesions in fundusimages, since we only had a singular grade for diabetic retinopathy per image,instead of multiple grades per images. Furthermore, the original study misseddetails regarding hyper-parameter settings for training and validation. Theoriginal study may also have used image quality grades as input for trainingthe network. We believe that deep learning algorithms should be easily replicated, andthat ideally source code should be published so that other researchers canconfirm the results of the experiments. Our source code and instructions forrunning the replication are available at:


Introduction (beta)



Conclusion (beta)