Maximum Entropy Inverse Reinforcement Learning for Mean Field Games

  • 2021-04-29 21:03:49
  • Yang Chen, Jiamou Liu, Bakhadyr Khoussainov
  • 1

Abstract

Mean field games (MFG) facilitate the otherwise intractable reinforcementlearning (RL) in large-scale multi-agent systems (MAS), through reducinginterplays among agents to those between a representative individual agent andthe mass of the population. While, RL agents are notoriously prone tounexpected behaviours due to reward mis-specification. This problem isexacerbated by an expanding scale of MAS. Inverse reinforcement learning (IRL)provides a framework to automatically acquire proper reward functions fromexpert demonstrations. Extending IRL to MFG, however, is challenging due to thecomplex notion of mean-field-type equilibria and the coupling betweenagent-level and population-level dynamics. To this end, we propose mean fieldinverse reinforcement learning (MFIRL), a novel model-free IRL framework forMFG. We derive the algorithm based on a new equilibrium concept thatincorporates entropy regularization, and the maximum entropy IRL framework.Experimental results on simulated environments demonstrate that MFIRL is sampleefficient and can accurately recover the ground-truth reward functions,compared to the state-of-the-art method.

 

Quick Read (beta)

loading the full paper ...