This work explores the large-scale multi-agent communication mechanism undera multi-agent reinforcement learning (MARL) setting. We summarize the generalcategories of topology for communication structures in MARL literature, whichare often manually specified. Then we propose a novel framework termed asLearning Structured Communication (LSC) by using a more flexible and efficientcommunication topology. Our framework allows for adaptive agent grouping toform different hierarchical formations over episodes, which is generated by anauxiliary task combined with a hierarchical routing protocol. Given each formedtopology, a hierarchical graph neural network is learned to enable effectivemessage information generation and propagation among inter- and intra-groupcommunications. In contrast to existing communication mechanisms, our methodhas an explicit while learnable design for hierarchical communication.Experiments on challenging tasks show the proposed LSC enjoys highcommunication efficiency, scalability, and global cooperation capability.