Multi-agent reinforcement learning (MARL) has been gaining extensiveattention from academia and industries in the past few decades. One of thefundamental problems in MARL is how to evaluate different approachescomprehensively. Most existing MARL methods are evaluated in either video gamesor simplistic simulated scenarios. It remains unknown how these methods performin real-world scenarios, especially multi-robot systems. This paper introducesa scalable emulation platform for multi-robot reinforcement learning (MRRL)called SMART to meet this need. Precisely, SMART consists of two components: 1)a simulation environment that provides a variety of complex interactionscenarios for training and 2) a real-world multi-robot system for realisticperformance evaluation. Besides, SMART offers agent-environment APIs that areplug-and-play for algorithm implementation. To illustrate the practicality ofour platform, we conduct a case study on the cooperative driving lane changescenario. Building off the case study, we summarize several unique challengesof MRRL, which are rarely considered previously. Finally, we open-source thesimulation environments, associated benchmark tasks, and state-of-the-artbaselines to encourage and empower MRRL research.