Abstract
Explainable Artificial Intelligence (XAI) has emerged as a pillar ofTrustworthy AI and aims to bring transparency in complex models that are opaqueby nature. Despite the benefits of incorporating explanations in models, anurgent need is found in addressing the privacy concerns of providing thisadditional information to end users. In this article, we conduct a scopingreview of existing literature to elicit details on the conflict between privacyand explainability. Using the standard methodology for scoping review, weextracted 57 articles from 1,943 studies published from January 2019 toDecember 2024. The review addresses 3 research questions to present readerswith more understanding of the topic: (1) what are the privacy risks ofreleasing explanations in AI systems? (2) what current methods have researchersemployed to achieve privacy preservation in XAI systems? (3) what constitutes aprivacy preserving explanation? Based on the knowledge synthesized from theselected studies, we categorize the privacy risks and preservation methods inXAI and propose the characteristics of privacy preserving explanations to aidresearchers and practitioners in understanding the requirements of XAI that isprivacy compliant. Lastly, we identify the challenges in balancing privacy withother system desiderata and provide recommendations for achieving privacypreserving XAI. We expect that this review will shed light on the complexrelationship of privacy and explainability, both being the fundamentalprinciples of Trustworthy AI.