Quantized Reinforcement Learning (QUARL)

  • 2019-10-02 16:22:29
  • Srivatsan Krishnan, Sharad Chitlangia, Maximilian Lam, Zishen Lam, Aleksandra Faust, Vijay Janapa Reddi
  • 4

Abstract

Recent work has shown that quantization can help reduce the memory, compute,and energy demands of deep neural networks without significantly harming theirquality. However, whether these prior techniques, applied traditionally toimage-based models, work with the same efficacy to the sequential decisionmaking process in reinforcement learning remains an unanswered question. Toaddress this void, we conduct the first comprehensive empirical study thatquantifies the effects of quantization on various deep reinforcement learningpolicies with the intent to reduce their computational resource demands. Weapply techniques such as post-training quantization and quantization awaretraining to a spectrum of reinforcement learning tasks (such as Pong, Breakout,BeamRider and more) and training algorithms (such as PPO, A2C, DDPG, and DQN).Across this spectrum of tasks and learning algorithms, we show that policiescan be quantized to 6-8 bits of precision without loss of accuracy. We alsoshow that certain tasks and reinforcement learning algorithms yield policiesthat are more difficult to quantize due to their effect of widening the models'distribution of weights and that quantization aware training consistentlyimproves results over post-training quantization and oftentimes even over thefull precision baseline. Finally, we demonstrate real-world applications ofquantization for reinforcement learning. We use half-precision training totrain a Pong model 50% faster, and we deploy a quantized reinforcement learningbased navigation policy to an embedded system, achieving an 18$\times$ speedupand a 4$\times$ reduction in memory usage over an unquantized policy.

 

Quick Read (beta)

loading the full paper ...