QHD: A brain-inspired hyperdimensional reinforcement learning algorithm

  • 2022-05-14 06:50:54
  • Yang Ni, Danny Abraham, Mariam Issa, Yeseong Kim, Pietro Mecati, Mohsen Imani
  • 1

Abstract

Reinforcement Learning (RL) has opened up new opportunities to solve a widerange of complex decision-making tasks. However, modern RL algorithms, e.g.,Deep Q-Learning, are based on deep neural networks, putting high computationalcosts when running on edge devices. In this paper, we propose QHD, aHyperdimensional Reinforcement Learning, that mimics brain properties towardrobust and real-time learning. QHD relies on a lightweight brain-inspired modelto learn an optimal policy in an unknown environment. We first develop a novelmathematical foundation and encoding module that maps state-action space intohigh-dimensional space. We accordingly develop a hyperdimensional regressionmodel to approximate the Q-value function. The QHD-powered agent makesdecisions by comparing Q-values of each possible action. We evaluate the effectof the different RL training batch sizes and local memory capacity on the QHDquality of learning. Our QHD is also capable of online learning with tiny localmemory capacity, which can be as small as the training batch size. QHD providesreal-time learning by further decreasing the memory capacity and the batchsize. This makes QHD suitable for highly-efficient reinforcement learning inthe edge environment, where it is crucial to support online and real-timelearning. Our solution also supports a small experience replay batch size thatprovides 12.3 times speedup compared to DQN while ensuring minimal qualityloss. Our evaluation shows QHD capability for real-time learning, providing34.6 times speedup and significantly better quality of learning thanstate-of-the-art deep RL algorithms.

 

Quick Read (beta)

loading the full paper ...