Abstract
In the multi-objective reinforcement learning (MORL) paradigm, the relativeimportance of each environment objective is often unknown prior to training, soagents must learn to specialize their behavior to optimize differentcombinations of environment objectives that are specified post-training. Theseare typically linear combinations, so the agent is effectively parameterized bya weight vector that describes how to balance competing environment objectives.However, many real world behaviors require non-linear combinations ofobjectives. Additionally, the conversion between desired behavior andweightings is often unclear. In this work, we explore the use of a language based on propositional logicwith quantitative semantics--in place of weight vectors--for specifyingnon-linear behaviors in an interpretable way. We use a recurrent encoder toencode logical combinations of objectives, and train a MORL agent to generalizeover these encodings. We test our agent in several grid worlds with variousobjectives and show that our agent can generalize to many never-before-seenspecifications with performance comparable to single policy baseline agents. Wealso demonstrate our agent's ability to generate meaningful policies whenpresented with novel specifications and quickly specialize to novelspecifications.