Abstract
Deep Reinforcement Learning (RL) can yield capable agents and controlpolicies in several domains but is commonly plagued by prohibitively longtraining times. Additionally, in the case of continuous control problems, theapplicability of learned policies on real-world embedded devices is limited dueto the lack of real-time guarantees and portability of existing libraries. Toaddress these challenges, we present RLtools, a dependency-free, header-only,pure C++ library for deep supervised and reinforcement learning. Its novelarchitecture allows RLtools to be used on a wide variety of platforms, from HPCclusters over workstations and laptops to smartphones, smartwatches, andmicrocontrollers. Specifically, due to the tight integration of the RLalgorithms with simulation environments, RLtools can solve popular RL problemsup to 76 times faster than other popular RL frameworks. We also benchmark theinference on a diverse set of microcontrollers and show that in most cases ouroptimized implementation is by far the fastest. Finally, RLtools enables thefirst-ever demonstration of training a deep RL algorithm directly on amicrocontroller, giving rise to the field of TinyRL. The source code as well asdocumentation and live demos are available through our project page athttps://rl.tools.