Abstract
Reinforcement learning (RL) agents make decisions using nothing butobservations from the environment, and consequently, heavily rely on therepresentations of those observations. Though some recent breakthroughs haveused vector-based categorical representations of observations, often referredto as discrete representations, there is little work explicitly assessing thesignificance of such a choice. In this work, we provide a thorough empiricalinvestigation of the advantages of representing observations as vectors ofcategorical values within the context of reinforcement learning. We performevaluations on world-model learning, model-free RL, and ultimately continual RLproblems, where the benefits best align with the needs of the problem setting.We find that, when compared to traditional continuous representations, worldmodels learned over discrete representations accurately model more of the worldwith less capacity, and that agents trained with discrete representations learnbetter policies with less data. In the context of continual RL, these benefitstranslate into faster adapting agents. Additionally, our analysis suggests thatthe observed performance improvements can be attributed to the informationcontained within the latent vectors and potentially the encoding of thediscrete representation itself.