Abstract
Offline meta-reinforcement learning aims to equip agents with the ability torapidly adapt to new tasks by training on data from a set of different tasks.Context-based approaches utilize a history of state-action-reward transitions-- referred to as the context -- to infer representations of the current task,and then condition the agent, i.e., the policy and value function, on the taskrepresentations. Intuitively, the better the task representations capture theunderlying tasks, the better the agent can generalize to new tasks.Unfortunately, context-based approaches suffer from distribution mismatch, asthe context in the offline data does not match the context at test time,limiting their ability to generalize to the test tasks. This leads to the taskrepresentations overfitting to the offline training data. Intuitively, the taskrepresentations should be independent of the behavior policy used to collectthe offline data. To address this issue, we approximately minimize the mutualinformation between the distribution over the task representations and behaviorpolicy by maximizing the entropy of behavior policy conditioned on the taskrepresentations. We validate our approach in MuJoCo environments, showing thatcompared to baselines, our task representations more faithfully represent theunderlying tasks, leading to outperforming prior methods in bothin-distribution and out-of-distribution tasks.