Abstract
Deep reinforcement learning (DRL) has shown success in diverse domains suchas robotics, computer games, and recommendation systems. However, like anyother software system, DRL-based software systems are susceptible to faultsthat pose unique challenges for debugging and diagnosing. These faults oftenresult in unexpected behavior without explicit failures and error messages,making debugging difficult and time-consuming. Therefore, automating themonitoring and diagnosis of DRL systems is crucial to alleviate the burden ondevelopers. In this paper, we propose RLExplorer, the first fault diagnosisapproach for DRL-based software systems. RLExplorer automatically monitorstraining traces and runs diagnosis routines based on properties of the DRLlearning dynamics to detect the occurrence of DRL-specific faults. It then logsthe results of these diagnoses as warnings that cover theoretical concepts,recommended practices, and potential solutions to the identified faults. Weconducted two sets of evaluations to assess RLExplorer. Our first evaluation offaulty DRL samples from Stack Overflow revealed that our approach caneffectively diagnose real faults in 83% of the cases. Our second evaluation ofRLExplorer with 15 DRL experts/developers showed that (1) RLExplorer couldidentify 3.6 times more defects than manual debugging and (2) RLExplorer iseasily integrated into DRL applications.