Abstract
Reinforcement learning (RL) training involves the deep nesting of highlyirregular computation patterns, each of which typically exhibits opportunitiesfor distributed computation. Current RL libraries offer parallelism at thelevel of the entire program, coupling all algorithm components together andmaking existing implementations difficult to scale, combine, and reuse. Weargue for distributing RL components in a composable way by adapting algorithmsfor top-down hierarchical control, thereby encapsulating parallelism andresource requirements within short-running compute tasks. We demonstrate this principle by building RLlib on top of a task-basedframework and show that we can implement a wide range of state-of-the artalgorithms on top of a small set of general abstractions. These abstractionsare key to composability and reuse in RLlib and do not come at the cost ofperformance---in our experiments, RLlib matches or exceeds the performance ofhighly optimized reference implementations. Ray RLlib is available as part ofRay at https://github.com/ray-project/ray/.