Abstract
Offline Reinforcement Learning (ORL) is a promising approach to reduce thehigh sample complexity of traditional Reinforcement Learning (RL) byeliminating the need for continuous environmental interactions. ORL exploits adataset of pre-collected transitions and thus expands the range of applicationof RL to tasks in which the excessive environment queries increase trainingtime and decrease efficiency, such as in modern AAA games. This paperintroduces OfflineMania a novel environment for ORL research. It is inspired bythe iconic TrackMania series and developed using the Unity 3D game engine. Theenvironment simulates a single-agent racing game in which the objective is tocomplete the track through optimal navigation. We provide a variety of datasetsto assess ORL performance. These datasets, created from policies of varyingability and in different sizes, aim to offer a challenging testbed foralgorithm development and evaluation. We further establish a set of baselinesfor a range of Online RL, ORL, and hybrid Offline to Online RL approaches usingour environment.