Abstract
In a sequential decision-making problem, off-policy evaluation estimates theexpected cumulative reward of a target policy using logged trajectory datagenerated from a different behavior policy, without execution of the targetpolicy. Reinforcement learning in high-stake environments, such as healthcareand education, is often limited to off-policy settings due to safety or ethicalconcerns, or inability of exploration. Hence it is imperative to quantify theuncertainty of the off-policy estimate before deployment of the target policy.In this paper, we propose a novel framework that provides robust and optimisticcumulative reward estimates using one or multiple logged trajectories data.Leveraging methodologies from distributionally robust optimization, we showthat with proper selection of the size of the distributional uncertainty set,these estimates serve as confidence bounds with non-asymptotic and asymptoticguarantees under stochastic or adversarial environments. Our results are alsogeneralized to batch reinforcement learning and are supported by empiricalanalysis.