Abstract
Evolutionary Reinforcement Learning (EvoRL) has emerged as a promisingapproach to overcoming the limitations of traditional reinforcement learning(RL) by integrating the Evolutionary Computation (EC) paradigm with RL.However, the population-based nature of EC significantly increasescomputational costs, thereby restricting the exploration of algorithmic designchoices and scalability in large-scale settings. To address this challenge, weintroduce $\texttt{$\textbf{EvoRL}$}$, the first end-to-end EvoRL frameworkoptimized for GPU acceleration. The framework executes the entire trainingpipeline on accelerators, including environment simulations and EC processes,leveraging hierarchical parallelism through vectorization and compilationtechniques to achieve superior speed and scalability. This design enables theefficient training of large populations on a single machine. In addition to itsperformance-oriented design, $\texttt{$\textbf{EvoRL}$}$ offers a comprehensiveplatform for EvoRL research, encompassing implementations of traditional RLalgorithms (e.g., A2C, PPO, DDPG, TD3, SAC), Evolutionary Algorithms (e.g.,CMA-ES, OpenES, ARS), and hybrid EvoRL paradigms such as Evolutionary-guided RL(e.g., ERL, CEM-RL) and Population-Based AutoRL (e.g., PBT). The framework'smodular architecture and user-friendly interface allow researchers toseamlessly integrate new components, customize algorithms, and conduct fairbenchmarking and ablation studies. The project is open-source and available at:https://github.com/EMI-Group/evorl.