Abstract
We consider offline reinforcement learning (RL) methods in possiblynonstationary environments. Many existing RL algorithms in the literature relyon the stationarity assumption that requires the system transition and thereward function to be constant over time. However, the stationarity assumptionis restrictive in practice and is likely to be violated in a number ofapplications, including traffic signal control, robotics and mobile health. Inthis paper, we develop a consistent procedure to test the nonstationarity ofthe optimal Q-function based on pre-collected historical data, withoutadditional online data collection. Based on the proposed test, we furtherdevelop a sequential change point detection method that can be naturallycoupled with existing state-of-the-art RL methods for policy optimization innonstationary environments. The usefulness of our method is illustrated bytheoretical results, simulation studies, and a real data example from the 2018Intern Health Study. A Python implementation of the proposed procedure isavailable at https://github.com/limengbinggz/CUSUM-RL.