Abstract
Existing multi-agent reinforcement learning methods are limited typically toa small number of agents. When the agent number increases largely, the learningbecomes intractable due to the curse of the dimensionality and the exponentialgrowth of agent interactions. In this paper, we present Mean FieldReinforcement Learning where the interactions within the population of agentsare approximated by those between a single agent and the average effect fromthe overall population or neighboring agents; the interplay between the twoentities is mutually reinforced: the learning of the individual agent's optimalpolicy depends on the dynamics of the population, while the dynamics of thepopulation change according to the collective patterns of the individualpolicies. We develop practical mean field Q-learning and mean fieldActor-Critic algorithms and analyze the convergence of the solution to Nashequilibrium. Experiments on Gaussian squeeze, Ising model, and battle gamesjustify the learning effectiveness of our mean field approaches. In addition,we report the first result to solve the Ising model via model-freereinforcement learning methods.