Model Imitation for Model-Based Reinforcement Learning

  • 2019-10-02 06:19:21
  • Yueh-Hua Wu, Ting-Han Fan, Peter J. Ramadge, Hao Su
  • 0

Abstract

Model-based reinforcement learning (MBRL) aims to learn a dynamic model toreduce the number of interactions with real-world environments. However, due toestimation error, rollouts in the learned model, especially those of longhorizon, fail to match the ones in real-world environments. This mismatchinghas seriously impacted the sample complexity of MBRL. The phenomenon can beattributed to the fact that previous works employ supervised learning to learnthe one-step transition models, which has inherent difficulty ensuring thematching of distributions from multi-step rollouts. Based on the claim, wepropose to learn the synthesized model by matching the distributions ofmulti-step rollouts sampled from the synthesized model and the real ones viaWGAN. We theoretically show that matching the two can minimize the differenceof cumulative rewards between the real transition and the learned one. Ourexperiments also show that the proposed model imitation method outperforms thestate-of-the-art in terms of sample complexity and average return.

 

Quick Read (beta)

loading the full paper ...