Abstract
Deep reinforcement learning has recently shown many impressive successes.However, one major obstacle towards applying such methods to real-worldproblems is their lack of data-efficiency. To this end, we propose theBottleneck Simulator: a model-based reinforcement learning method whichcombines a learned, factorized transition model of the environment with rolloutsimulations to learn an effective policy from few examples. The learnedtransition model employs an abstract, discrete (bottleneck) state, whichincreases sample efficiency by reducing the number of model parameters and byexploiting structural properties of the environment. We provide a mathematicalanalysis of the Bottleneck Simulator in terms of fixed points of the learnedpolicy, which reveals how performance is affected by four distinct sources oferror: an error related to the abstract space structure, an error related tothe transition model estimation variance, an error related to the transitionmodel estimation bias, and an error related to the transition model class bias.Finally, we evaluate the Bottleneck Simulator on two natural languageprocessing tasks: a text adventure game and a real-world, complex dialogueresponse selection task. On both tasks, the Bottleneck Simulator yieldsexcellent performance beating competing approaches.