Abstract
Safe reinforcement learning (RL) requires the agent to finish a given taskwhile obeying specific constraints. Giving constraints in natural language formhas great potential for practical scenarios due to its flexible transfercapability and accessibility. Previous safe RL methods with natural languageconstraints typically need to design cost functions manually for eachconstraint, which requires domain expertise and lacks flexibility. In thispaper, we harness the dual role of text in this task, using it not only toprovide constraint but also as a training signal. We introduce theTrajectory-level Textual Constraints Translator (TTCT) to replace the manuallydesigned cost function. Our empirical results demonstrate that TTCT effectivelycomprehends textual constraint and trajectory, and the policies trained by TTCTcan achieve a lower violation rate than the standard cost function. Extrastudies are conducted to demonstrate that the TTCT has zero-shot transfercapability to adapt to constraint-shift environments.