Abstract
Existing modular Reinforcement Learning (RL) architectures are generallybased on reusable components, also allowing for "plug-and-play" integration.However, these modules are homogeneous in nature - in fact, they essentiallyprovide policies obtained via RL through the maximization of individual rewardfunctions. Consequently, such solutions still lack the ability to integrate andprocess multiple types of information (i.e., heterogeneous knowledgerepresentations), such as rules, sub-goals, and skills from various sources. Inthis paper, we discuss several practical examples of heterogeneous knowledgeand propose Augmented Modular Reinforcement Learning (AMRL) to address theselimitations. Our framework uses a selector to combine heterogeneous modules andseamlessly incorporate different types of knowledge representations andprocessing mechanisms. Our results demonstrate the performance and efficiencyimprovements, also in terms of generalization, that can be achieved byaugmenting traditional modular RL with heterogeneous knowledge sources andprocessing mechanisms. Finally, we examine the safety, robustness, andinterpretability issues stemming from the introduction of knowledgeheterogeneity.