Abstract
This paper investigates how deep multi-agent reinforcement learning canenable the scalable and privacy-preserving coordination of residential energyflexibility. The coordination of distributed resources such as electricvehicles and heating will be critical to the successful integration of largeshares of renewable energy in our electricity grid and, thus, to help mitigateclimate change. The pre-learning of individual reinforcement learning policiescan enable distributed control with no sharing of personal data required duringexecution. However, previous approaches for multi-agent reinforcementlearning-based distributed energy resources coordination impose an ever greatertraining computational burden as the size of the system increases. We thereforeadopt a deep multi-agent actor-critic method which uses a \emph{centralised butfactored critic} to rehearse coordination ahead of execution. Results show thatcoordination is achieved at scale, with minimal information and communicationinfrastructure requirements, no interference with daily activities, and privacyprotection. Significant savings are obtained for energy users, the distributionnetwork and greenhouse gas emissions. Moreover, training times are nearly 40times shorter than with a previous state-of-the-art reinforcement learningapproach without the factored critic for 30 homes.