Abstract
Reinforcement learning (RL) is a machine learning approach that trains agentsto maximize cumulative rewards through interactions with environments. Theintegration of RL with deep learning has recently resulted in impressiveachievements in a wide range of challenging tasks, including board games,arcade games, and robot control. Despite these successes, there remain severalcrucial challenges, including brittle convergence properties caused bysensitive hyperparameters, difficulties in temporal credit assignment with longtime horizons and sparse rewards, a lack of diverse exploration, especially incontinuous search space scenarios, difficulties in credit assignment inmulti-agent reinforcement learning, and conflicting objectives for rewards.Evolutionary computation (EC), which maintains a population of learning agents,has demonstrated promising performance in addressing these limitations. Thisarticle presents a comprehensive survey of state-of-the-art methods forintegrating EC into RL, referred to as evolutionary reinforcement learning(EvoRL). We categorize EvoRL methods according to key research fields in RL,including hyperparameter optimization, policy search, exploration, rewardshaping, meta-RL, and multi-objective RL. We then discuss future researchdirections in terms of efficient methods, benchmarks, and scalable platforms.This survey serves as a resource for researchers and practitioners interestedin the field of EvoRL, highlighting the important challenges and opportunitiesfor future research. With the help of this survey, researchers andpractitioners can develop more efficient methods and tailored benchmarks forEvoRL, further advancing this promising cross-disciplinary research field.