Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

  • 2020-03-19 16:51:51
  • Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson
  • 2

Abstract

In many real-world settings, a team of agents must coordinate its behaviourwhile acting in a decentralised fashion. At the same time, it is often possibleto train the agents in a centralised fashion where global state information isavailable and communication constraints are lifted. Learning jointaction-values conditioned on extra state information is an attractive way toexploit centralised learning, but the best strategy for then extractingdecentralised policies is unclear. Our solution is QMIX, a novel value-basedmethod that can train decentralised policies in a centralised end-to-endfashion. QMIX employs a mixing network that estimates joint action-values as amonotonic combination of per-agent values. We structurally enforce that thejoint-action value is monotonic in the per-agent values, through the use ofnon-negative weights in the mixing network, which guarantees consistencybetween the centralised and decentralised policies. To evaluate the performanceof QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a newbenchmark for deep multi-agent reinforcement learning. We evaluate QMIX on achallenging set of SMAC scenarios and show that it significantly outperformsexisting multi-agent reinforcement learning methods.

 

Quick Read (beta)

loading the full paper ...