Abstract
Compositional knowledge representations in reinforcement learning (RL)facilitate modular, interpretable, and safe task specifications. However,generating compositional models requires the characterization of minimalassumptions for the robustness of the compositionality feature, especially inthe case of functional decompositions. Using a categorical point of view, wedevelop a knowledge representation framework for a compositional theory of RL.Our approach relies on the theoretical study of the category $\mathsf{MDP}$,whose objects are Markov decision processes (MDPs) acting as models of tasks.The categorical semantics models the compositionality of tasks through theapplication of pushout operations akin to combining puzzle pieces. As apractical application of these pushout operations, we introduce zig-zagdiagrams that rely on the compositional guarantees engendered by the category$\mathsf{MDP}$. We further prove that properties of the category $\mathsf{MDP}$unify concepts, such as enforcing safety requirements and exploitingsymmetries, generalizing previous abstraction theories for RL.