Abstract
In multi-agent reinforcement learning, centralised policies can only beexecuted if agents have access to either the global state or an instantaneouscommunication channel. An alternative approach that circumvents this limitationis to use centralised training of a set of decentralised policies. However,such policies severely limit the agents' ability to coordinate. We proposemulti-agent common knowledge reinforcement learning (MACKRL), which strikes amiddle ground between these two extremes. Our approach is based on the insightthat, even in partially observable settings, subsets of agents often have somecommon knowledge that they can exploit to coordinate their behaviour. Commonknowledge can arise, e.g., if all agents can reliably observe things in theirown field of view and know the field of view of other agents. Using thisadditional information, it is possible to find a centralised policy thatconditions only on agents' common knowledge and that can be executed in adecentralised fashion. A resulting challenge is then to determine at what levelagents should coordinate. While the common knowledge shared among all agentsmay not contain much valuable information, there may be subgroups of agentsthat share common knowledge useful for coordination. MACKRL addresses thischallenge using a hierarchical approach: at each level, a controller can eitherselect a joint action for the agents in a given subgroup, or propose apartition of the agents into smaller subgroups whose actions are then selectedby controllers at the next level. While action selection involves samplinghierarchically, learning updates are based on the probability of the jointaction, calculated by marginalising across the possible decisions of thehierarchy. We show promising results on both a proof-of-concept matrix game anda multi-agent version of StarCraft II Micromanagement.