Abstract
Reinforcement learning (RL) algorithms aim to balance exploiting the currentbest strategy with exploring new options that could lead to higher rewards.Most common RL algorithms use undirected exploration, i.e., select randomsequences of actions. Exploration can also be directed using intrinsic rewards,such as curiosity or model epistemic uncertainty. However, effectivelybalancing task and intrinsic rewards is challenging and often task-dependent.In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic andextrinsic exploration. MaxInfoRL steers exploration towards informativetransitions, by maximizing intrinsic rewards such as the information gain aboutthe underlying task. When combined with Boltzmann exploration, this approachnaturally trades off maximization of the value function with that of theentropy over states, rewards, and actions. We show that our approach achievessublinear regret in the simplified setting of multi-armed bandits. We thenapply this general formulation to a variety of off-policy model-free RL methodsfor continuous state-action spaces, yielding novel algorithms that achievesuperior performance across hard exploration problems and complex scenariossuch as visual control tasks.