Abstract
In deep reinforcement learning (RL) systems, abnormal states pose significantrisks by potentially triggering unpredictable behaviors and unsafe actions,thus impeding the deployment of RL systems in real-world scenarios. It iscrucial for reliable decision-making systems to have the capability to cast analert whenever they encounter unfamiliar observations that they are notequipped to handle. In this paper, we propose a novel Mahalanobisdistance-based (MD) anomaly detection framework, called \textit{MDX}, for deepRL algorithms. MDX simultaneously addresses random, adversarial, andout-of-distribution (OOD) state outliers in both offline and online settings.It utilizes Mahalanobis distance within class-conditional distributions foreach action and operates within a statistical hypothesis testing frameworkunder the Gaussian assumption. We further extend it to robust anddistribution-free versions by incorporating Robust MD and conformal inferencetechniques. Through extensive experiments on classical control environments,Atari games, and autonomous driving scenarios, we demonstrate the effectivenessof our MD-based detection framework. MDX offers a simple, unified, andpractical anomaly detection tool for enhancing the safety and reliability of RLsystems in real-world applications.