Abstract
As reinforcement learning agents become increasingly integrated into complex,real-world environments, designing for safety becomes a critical consideration.We specifically focus on researching scenarios where agents can cause undesiredside effects while executing a policy on a primary task. Since one can definemultiple tasks for a given environment dynamics, there are two importantchallenges. First, we need to abstract the concept of safety that appliesbroadly to that environment independent of the specific task being executed.Second, we need a mechanism for the abstracted notion of safety to modulate theactions of agents executing different policies to minimize their side-effects.In this work, we propose Safety Aware Reinforcement Learning (SARL) - aframework where a virtual safe agent modulates the actions of a mainreward-based agent to minimize side effects. The safe agent learns atask-independent notion of safety for a given environment. The main agent isthen trained with a regularization loss given by the distance between thenative action probabilities of the two agents. Since the safe agent effectivelyabstracts a task-independent notion of safety via its action probabilities, itcan be ported to modulate multiple policies solving different tasks within thegiven environment without further training. We contrast this with solutionsthat rely on task-specific regularization metrics and test our framework on theSafeLife Suite, based on Conway's Game of Life, comprising a number of complextasks in dynamic environments. We show that our solution is able to match theperformance of solutions that rely on task-specific side-effect penalties onboth the primary and safety objectives while additionally providing the benefitof generalizability and portability.