Abstract
We introduce a new generative model for human planning under the BayesianInverse Reinforcement Learning (BIRL) framework which takes into account thefact that humans often plan using hierarchical strategies. We describe theBayesian Inverse Hierarchical RL (BIHRL) algorithm for inferring the values ofhierarchical planners, and use an illustrative toy model to show that BIHRLretains accuracy where standard BIRL fails. Furthermore, BIHRL is able toaccurately predict the goals of `Wikispeedia' game players, with inclusion ofhierarchical structure in the model resulting in a large boost in accuracy. Weshow that BIHRL is able to significantly outperform BIRL even when we only havea weak prior on the hierarchical structure of the plans available to the agent,and discuss the significant challenges that remain for scaling up thisframework to more realistic settings.