Abstract
Despite the myriad peer-reviewed papers demonstrating novel ArtificialIntelligence (AI)-based solutions to COVID-19 challenges during the pandemic,few have made significant clinical impact. The impact of artificialintelligence during the COVID-19 pandemic was greatly limited by lack of modeltransparency. This systematic review examines the use of Explainable ArtificialIntelligence (XAI) during the pandemic and how its use could overcome barriersto real-world success. We find that successful use of XAI can improve modelperformance, instill trust in the end-user, and provide the value needed toaffect user decision-making. We introduce the reader to common XAI techniques,their utility, and specific examples of their application. Evaluation of XAIresults is also discussed as an important step to maximize the value ofAI-based clinical decision support systems. We illustrate the classical,modern, and potential future trends of XAI to elucidate the evolution of novelXAI techniques. Finally, we provide a checklist of suggestions during theexperimental design process supported by recent publications. Common challengesduring the implementation of AI solutions are also addressed with specificexamples of potential solutions. We hope this review may serve as a guide toimprove the clinical impact of future AI-based solutions.