A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning

  • 2022-01-01 06:42:51
  • Yuxing Wang, Tiantian Zhang, Yongzhe Chang, Bin Liang, Xueqian Wang, Bo Yuan
  • 0


The integration of Reinforcement Learning (RL) and Evolutionary Algorithms(EAs) aims at simultaneously exploiting the sample efficiency as well as thediversity and robustness of the two paradigms. Recently, hybrid learningframeworks based on this principle have achieved great success in variouschallenging robot control tasks. However, in these methods, policies from thegenetic population are evaluated via interactions with the real environments,limiting their applicability in computationally expensive problems. In thiswork, we propose Surrogate-assisted Controller (SC), a novel and efficientmodule that can be integrated into existing frameworks to alleviate thecomputational burden of EAs by partially replacing the expensive policyevaluation. The key challenge in applying this module is to prevent theoptimization process from being misled by the possible false minima introducedby the surrogate. To address this issue, we present two strategies for SC tocontrol the workflow of hybrid frameworks. Experiments on six continuouscontrol tasks from the OpenAI Gym platform show that SC can not onlysignificantly reduce the cost of fitness evaluations, but also boost theperformance of the original hybrid frameworks with collaborative learning andevolutionary processes.


Quick Read (beta)

loading the full paper ...