Abstract
Safety controllers is widely used to achieve safe reinforcement learning.Most methods that apply a safety controller are using handcrafted safetyconstraints to construct the safety controller. However, when the environmentdynamics are sophisticated, handcrafted safety constraints become unavailable.Therefore, it worth to research on constructing safety controllers by learningalgorithms. We propose a three-stage architecture for safe reinforcementlearning, namely TU-Recovery Architecture. A safety critic and a recoverypolicy is learned before task training. They form a safety controller to ensuresafety in task training. Then a phenomenon induced by disagreement between taskpolicy and recovery policy, called adversarial phenomenon, which reduceslearning efficiency and model performance, is described. Auxiliary reward isproposed to mitigate adversarial phenomenon, while help the task policy tolearn to recover from high-risk states. A series of experiments are conductedin a robot navigation environment. Experiments demonstrate that TU-Recoveryoutperforms unconstrained counterpart in both reward gaining and constraintviolations during task training, and auxiliary reward further improveTU-Recovery in reward-to-cost ratio by significantly reduce constraintviolations.