Abstract
Multi-agent reinforcement learning (MARL) requires agents to explore within avast joint action space to find joint actions that lead to coordination.Existing value-based MARL algorithms commonly rely on random exploration, suchas $\epsilon$-greedy, to explore the environment which is not systematic andinefficient at identifying effective actions in multi-agent problems.Additionally, the concurrent training of the policies of multiple agents duringtraining can render the optimisation non-stationary. This can lead to unstablevalue estimates, highly variant gradients, and ultimately hinder coordinationbetween agents. To address these challenges, we propose ensemble valuefunctions for multi-agent exploration (EMAX). EMAX is a framework to seamlesslyextend value-based MARL algorithms. EMAX leverages an ensemble of valuefunctions for each agent to guide their exploration, reduce the variance oftheir optimisation, and makes their policies more robust to miscoordination.EMAX achieves these benefits by (1) systematically guiding the exploration ofagents with a UCB policy towards parts of the environment that require multipleagents to coordinate. (2) EMAX computes average value estimates across theensemble as target values to reduce the variance of gradients and makeoptimisation more stable. (3) During evaluation, EMAX selects actions followinga majority vote across the ensemble to reduce the likelihood ofmiscoordination. We first instantiate independent DQN with EMAX and evaluate itin 11 general-sum tasks with sparse rewards. We show that EMAX improves finalevaluation returns by 185% across all tasks. We then evaluate EMAX on top ofIDQN, VDN and QMIX in 21 common-reward tasks, and show that EMAX improvessample efficiency and final evaluation returns across all tasks over all threevanilla algorithms by 60%, 47%, and 538%, respectively.