Electric Vehicle (EV) has become a preferable choice in the moderntransportation system due to its environmental and energy sustainability.However, in many large cities, EV drivers often fail to find the proper spotsfor charging, because of the limited charging infrastructures and thespatiotemporally unbalanced charging demands. Indeed, the recent emergence ofdeep reinforcement learning provides great potential to improve the chargingexperience from various aspects over a long-term horizon. In this paper, wepropose a framework, named Multi-Agent Spatio-Temporal Reinforcement Learning(Master), for intelligently recommending public accessible charging stations byjointly considering various long-term spatiotemporal factors. Specifically, byregarding each charging station as an individual agent, we formulate thisproblem as a multi-objective multi-agent reinforcement learning task. We firstdevelop a multi-agent actor-critic framework with the centralized attentivecritic to coordinate the recommendation between geo-distributed agents.Moreover, to quantify the influence of future potential charging competition,we introduce a delayed access strategy to exploit the knowledge of futurecharging competition during training. After that, to effectively optimizemultiple learning objectives, we extend the centralized attentive critic tomulti-critics and develop a dynamic gradient re-weighting strategy toadaptively guide the optimization direction. Finally, extensive experiments ontwo real-world datasets demonstrate that Master achieves the best comprehensiveperformance compared with nine baseline approaches.