Abstract
Planning and reinforcement learning are two key approaches to sequentialdecision making. Multi-step approximate real-time dynamic programming, arecently successful algorithm class of which AlphaZero [Silver et al., 2018] isan example, combines both by nesting planning within a learning loop. However,the combination of planning and learning introduces a new question: how shouldwe balance time spend on planning, learning and acting? The importance of thistrade-off has not been explicitly studied before. We show that it is actuallyof key importance, with computational results indicating that we should neitherplan too long nor too short. Conceptually, we identify a new spectrum ofplanning-learning algorithms which ranges from exhaustive search (longplanning) to model-free RL (no planning), with optimal performance achievedmidway.