Abstract
Multi-agent deep reinforcement learning makes optimal decisions dependent onsystem states observed by agents, but any uncertainty on the observations maymislead agents to take wrong actions. The Mean-Field Actor-Critic reinforcementlearning (MFAC) is well-known in the multi-agent field since it can effectivelyhandle a scalability problem. However, it is sensitive to state perturbationsthat can significantly degrade the team rewards. This work proposes a RobustMean-field Actor-Critic reinforcement learning (RoMFAC) that has twoinnovations: 1) a new objective function of training actors, composed of a\emph{policy gradient function} that is related to the expected cumulativediscount reward on sampled clean states and an \emph{action loss function} thatrepresents the difference between actions taken on clean and adversarialstates; and 2) a repetitive regularization of the action loss, ensuring thetrained actors to obtain excellent performance. Furthermore, this work proposesa game model named a State-Adversarial Stochastic Game (SASG). Despite the Nashequilibrium of SASG may not exist, adversarial perturbations to states in theRoMFAC are proven to be defensible based on SASG. Experimental results showthat RoMFAC is robust against adversarial perturbations while maintaining itscompetitive performance in environments without perturbations.