Abstract
As AI systems become increasingly autonomous, reliably aligning theirdecision-making with human preferences is essential. Inverse reinforcementlearning (IRL) offers a promising approach to infer preferences fromdemonstrations. These preferences can then be used to produce an apprenticepolicy that performs well on the demonstrated task. However, in domains likeautonomous driving or robotics, where errors can have serious consequences, weneed not just good average performance but reliable policies with formalguarantees -- yet obtaining sufficient human demonstrations for reliabilityguarantees can be costly. Active IRL addresses this challenge by strategicallyselecting the most informative scenarios for human demonstration. We introducePAC-EIG, an information-theoretic acquisition function that directly targetsprobably-approximately-correct (PAC) guarantees for the learned policy --providing the first such theoretical guarantee for active IRL with noisy expertdemonstrations. Our method maximises information gain about the regret of theapprentice policy, efficiently identifying states requiring furtherdemonstration. We also present Reward-EIG as an alternative when learning thereward itself is the primary objective. Focusing on finite state-action spaces,we prove convergence bounds, illustrate failure modes of prior heuristicmethods, and demonstrate our method's advantages experimentally.