Abstract
Inverse reinforcement learning (IRL) is the problem of learning thepreferences of an agent from the observations of its behavior on a task. Whilethis problem has been well investigated, the related problem of {\em online}IRL---where the observations are incrementally accrued, yet the demands of theapplication often prohibit a full rerun of an IRL method---has receivedrelatively less attention. We introduce the first formal framework for onlineIRL, called incremental IRL (I2RL), and a new method that advances maximumentropy IRL with hidden variables, to this setting. Our formal analysis showsthat the new method has a monotonically improving performance with moredemonstration data, as well as probabilistically bounded error, both under fulland partial observability. Experiments in a simulated robotic application ofpenetrating a continuous patrol under occlusion shows the relatively improvedperformance and speed up of the new method and validates the utility of onlineIRL.