Abstract
Offline reinforcement learning (RL) extends the paradigm of classical RLalgorithms to purely learning from static datasets, without interacting withthe underlying environment during the learning process. A key challenge ofoffline RL is the instability of policy training, caused by the mismatchbetween the distribution of the offline data and the undiscounted stationarystate-action distribution of the learned policy. To avoid the detrimentalimpact of distribution mismatch, we regularize the undiscounted stationarydistribution of the current policy towards the offline data during the policyoptimization process. Further, we train a dynamics model to both implement thisregularization and better estimate the stationary distribution of the currentpolicy, reducing the error induced by distribution mismatch. On a wide range ofcontinuous-control offline RL datasets, our method indicates competitiveperformance, which validates our algorithm. The code is publicly available.