Abstract
While deep reinforcement learning (DRL) has led to numerous successes inrecent years, reproducing these successes can be extremely challenging. Onereproducibility challenge particularly relevant to DRL is nondeterminism in thetraining process, which can substantially affect the results. Motivated by thischallenge, we study the positive impacts of deterministic implementations ineliminating nondeterminism in training. To do so, we consider the particularcase of the deep Q-learning algorithm, for which we produce a deterministicimplementation by identifying and controlling all sources of nondeterminism inthe training process. One by one, we then allow individual sources ofnondeterminism to affect our otherwise deterministic implementation, andmeasure the impact of each source on the variance in performance. We find thatindividual sources of nondeterminism can substantially impact the performanceof agent, illustrating the benefits of deterministic implementations. Inaddition, we also discuss the important role of deterministic implementationsin achieving exact replicability of results.