Over the past decades, deep learning (DL) systems have achieved tremendoussuccess and gained great popularity in various applications, such asintelligent machines, image processing, speech processing, and medicaldiagnostics. Deep neural networks are the key driving force behind its recentsuccess, but still seem to be a magic black box lacking interpretability andunderstanding. This brings up many open safety and security issues withenormous and urgent demands on rigorous methodologies and engineering practicefor quality enhancement. A plethora of studies have shown that thestate-of-the-art DL systems suffer from defects and vulnerabilities that canlead to severe loss and tragedies, especially when applied to real-worldsafety-critical applications. In this paper, we perform a large-scale study andconstruct a paper repository of 223 relevant works to the quality assurance,security, and interpretation of deep learning. We, from a software qualityassurance perspective, pinpoint challenges and future opportunities towardsuniversal secure deep learning engineering. We hope this work and theaccompanied paper repository can pave the path for the software engineeringcommunity towards addressing the pressing industrial demand of secureintelligent applications.