Abstract
State-of-the-art reinforcement learning algorithms mostly rely on beingallowed to directly interact with their environment to collect millions ofobservations. This makes it hard to transfer their success to industrialcontrol problems, where simulations are often very costly or do not exist, andexploring in the real environment can potentially lead to catastrophic events.Recently developed, model-free, offline RL algorithms, can learn from a singledataset (containing limited exploration) by mitigating extrapolation error invalue functions. However, the robustness of the training process is stillcomparatively low, a problem known from methods using value functions. Toimprove robustness and stability of the learning process, we use dynamicsmodels to assess policy performance instead of value functions, resulting inMOOSE (MOdel-based Offline policy Search with Ensembles), an algorithm whichensures low model bias by keeping the policy within the support of the data. Wecompare MOOSE with state-of-the-art model-free, offline RL algorithms { BRAC,}BEAR and BCQ on the Industrial Benchmark and MuJoCo continuous control tasks interms of robust performance, and find that MOOSE outperforms its model-freecounterparts in almost all considered cases, often even by far.