Abstract
Deploying deep reinforcement learning in safety-critical settings requiresdeveloping algorithms that obey hard constraints during exploration. This papercontributes a first approach toward enforcing formal safety constraints onend-to-end policies with visual inputs. Our approach draws on recent advancesin object detection and automated reasoning for hybrid dynamical systems. Theapproach is evaluated on a novel benchmark that emphasizes the challenge ofsafely exploring in the presence of hard constraints. Our benchmark draws fromseveral proposed problem sets for safe learning and includes problems thatemphasize challenges such as reward signals that are not aligned with safetyconstraints. On each of these benchmark problems, our algorithm completelyavoids unsafe behavior while remaining competitive at optimizing for as muchreward as is safe. We also prove that our method of enforcing the safetyconstraints preserves all safe policies from the original environment.