Abstract

Many reinforcement learning (RL) environments consist of independent entitiesthat interact sparsely. In such environments, RL agents have only limitedinfluence over other entities in any particular situation. Our idea in thiswork is that learning can be efficiently guided by knowing when and what theagent can influence with its actions. To achieve this, we introduce a measureof \emph{situation-dependent causal influence} based on conditional mutualinformation and show that it can reliably detect states of influence. We thenpropose several ways to integrate this measure into RL algorithms to improveexploration and off-policy learning. All modified algorithms show strongincreases in data efficiency on robotic manipulation tasks.

 

Quick Read (beta)