Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

  • 2020-05-21 14:57:35
  • Shangtong Zhang, Bo Liu, Shimon Whiteson
  • 0

Abstract

We present a mean-variance policy iteration (MVPI) framework for risk-aversecontrol in a discounted infinite horizon MDP. MVPI enjoys great flexibility inthat any policy evaluation method and risk-neutral control method can bedropped in for risk-averse control off the shelf, in both on- and off-policysettings. We propose risk-averse TD3 as an example instantiating MVPI, whichoutperforms vanilla TD3 and many previous risk-averse control methods inchallenging Mujoco robot simulation tasks under a risk-aware performancemetric. This risk-averse TD3 is the first to introduce deterministic policiesand off-policy learning into risk-averse reinforcement learning, both of whichare key to the performance boost we show in Mujoco domains. MVPI adopts aper-step reward perspective (Bisi et al., 2019) for risk-averse control,instead of the commonly used total reward perspective.

 

Quick Read (beta)

loading the full paper ...