Sequential decision making, commonly formalized as Markov Decision Process(MDP) optimization, is a key challenge in artificial intelligence. Two keyapproaches to this problem are reinforcement learning (RL) and planning. Thispaper presents a survey of the integration of both fields, better known asmodel-based reinforcement learning. Model-based RL has two main steps. First,we systematically cover approaches to dynamics model learning, includingchallenges like dealing with stochasticity, uncertainty, partial observability,and temporal abstraction. Second, we present a systematic categorization ofplanning-learning integration, including aspects like: where to start planning,what budgets to allocate to planning and real data collection, how to plan, andhow to integrate planning in the learning and acting loop. After these two keysections, we also discuss the potential benefits of model-based RL, likeenhanced data efficiency, targeted exploration, and improved stability. Alongthe survey, we also draw connections to several related RL fields, likehierarchical RL and transfer, and other research disciplines, like behaviouralpsychology. Altogether, the survey presents a broad conceptual overview ofplanning-learning combinations for MDP optimization.