The analysis and control of large-population systems is of great interest todiverse areas of research and engineering, ranging from epidemiology overrobotic swarms to economics and finance. An increasingly popular and effectiveapproach to realizing sequential decision-making in multi-agent systems isthrough multi-agent reinforcement learning, as it allows for an automatic andmodel-free analysis of highly complex systems. However, the key issue ofscalability complicates the design of control and reinforcement learningalgorithms particularly in systems with large populations of agents. Whilereinforcement learning has found resounding empirical success in many scenarioswith few agents, problems with many agents quickly become intractable andnecessitate special consideration. In this survey, we will shed light oncurrent approaches to tractably understanding and analyzing large-populationsystems, both through multi-agent reinforcement learning and through adjacentareas of research such as mean-field games, collective intelligence, or complexnetwork theory. These classically independent subject areas offer a variety ofapproaches to understanding or modeling large-population systems, which may beof great use for the formulation of tractable MARL algorithms in the future.Finally, we survey potential areas of application for large-scale control andidentify fruitful future applications of learning algorithms in practicalsystems. We hope that our survey could provide insight and future directions tojunior and senior researchers in theoretical and applied sciences alike.