Abstract
Traditional reinforcement learning (RL) assumes the agents make decisionsbased on Markov decision processes (MDPs) with one-step transition models. Inmany real-world applications, such as energy management and stock investment,agents can access multi-step predictions of future states, which provideadditional advantages for decision making. However, multi-step predictions areinherently high-dimensional: naively embedding these predictions into an MDPleads to an exponential blow-up in state space and the curse of dimensionality.Moreover, existing RL theory provides few tools to analyze prediction-augmentedMDPs, as it typically works on one-step transition kernels and cannotaccommodate multi-step predictions with errors or partial action-coverage. Weaddress these challenges with three key innovations: First, we propose the\emph{Bayesian value function} to characterize the optimal prediction-awarepolicy tractably. Second, we develop a novel \emph{Bellman-Jensen Gap} analysison the Bayesian value function, which enables characterizing the value ofimperfect predictions. Third, we introduce BOLA (Bayesian Offline Learning withOnline Adaptation), a two-stage model-based RL algorithm that separates offlineBayesian value learning from lightweight online adaptation to real-timepredictions. We prove that BOLA remains sample-efficient even under imperfectpredictions. We validate our theory and algorithm on synthetic MDPs and areal-world wind energy storage control problem.