Abstract
Exploration is critical for good results in deep reinforcement learning andhas attracted much attention. However, existing multi-agent deep reinforcementlearning algorithms still use mostly noise-based techniques. Very recently,exploration methods that consider cooperation among multiple agents have beendeveloped. However, existing methods suffer from a common challenge: agentsstruggle to identify states that are worth exploring, and hardly coordinateexploration efforts toward those states. To address this shortcoming, in thispaper, we propose cooperative multi-agent exploration (CMAE): agents share acommon goal while exploring. The goal is selected from multiple projected statespaces via a normalized entropy-based technique. Then, agents are trained toreach this goal in a coordinated manner. We demonstrate that CMAE consistentlyoutperforms baselines on various tasks, including a sparse-reward version ofthe multiple-particle environment (MPE) and the Starcraft multi-agent challenge(SMAC).