Abstract
Recently, deep reinforcement learning (RL) methods have been appliedsuccessfully to multi-agent scenarios. Typically, these methods rely on aconcatenation of agent states to represent the information content required fordecentralized decision making. However, concatenation scales poorly to swarmsystems with a large number of homogeneous agents as it does not exploit thefundamental properties inherent to these systems: (i) the agents in the swarmare interchangeable and (ii) the exact number of agents in the swarm isirrelevant. Therefore, we propose a new state representation for deepmulti-agent RL based on mean embeddings of distributions. We treat the agentsas samples of a distribution and use the empirical mean embedding as input fora decentralized policy. We define different feature spaces of the meanembedding using histograms, radial basis functions and a neural network learnedend-to-end. We evaluate the representation on two well known problems from theswarm literature (rendezvous and pursuit evasion), in a globally and locallyobservable setup. For the local setup we furthermore introduce simplecommunication protocols. Of all approaches, the mean embedding representationusing neural network features enables the richest information exchange betweenneighboring agents facilitating the development of more complex collectivestrategies.