Abstract
Future Internet involves several emerging technologies such as 5G and beyond5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, andInternet of Things (IoTs). Moreover, future Internet becomes heterogeneous anddecentralized with a large number of involved network entities. Each entity mayneed to make its local decision to improve the network performance underdynamic and uncertain network environments. Standard learning algorithms suchas single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning(DRL) have been recently used to enable each network entity as an agent tolearn an optimal decision-making policy adaptively through interacting with theunknown environments. However, such an algorithm fails to model thecooperations or competitions among network entities, and simply treats otherentities as a part of the environment that may result in the non-stationarityissue. Multi-agent Reinforcement Learning (MARL) allows each network entity tolearn its optimal policy by observing not only the environments, but also otherentities' policies. As a result, MARL can significantly improve the learningefficiency of the network entities, and it has been recently used to solvevarious issues in the emerging networks. In this paper, we thus review theapplications of MARL in the emerging networks. In particular, we provide atutorial of MARL and a comprehensive survey of applications of MARL in nextgeneration Internet. In particular, we first introduce single-agent RL andMARL. Then, we review a number of applications of MARL to solve emerging issuesin future Internet. The issues consist of network access, transmit powercontrol, computation offloading, content caching, packet routing, trajectorydesign for UAV-aided networks, and network security issues.