Generalization is a central challenge for the deployment of reinforcementlearning (RL) systems in the real world. In this paper, we show that thesequential structure of the RL problem necessitates new approaches togeneralization beyond the well-studied techniques used in supervised learning.While supervised learning methods can generalize effectively without explicitlyaccounting for epistemic uncertainty, we show that, perhaps surprisingly, thisis not the case in RL. We show that generalization to unseen test conditionsfrom a limited number of training conditions induces implicit partialobservability, effectively turning even fully-observed MDPs into POMDPs.Informed by this observation, we recast the problem of generalization in RL assolving the induced partially observed Markov decision process, which we callthe epistemic POMDP. We demonstrate the failure modes of algorithms that do notappropriately handle this partial observability, and suggest a simpleensemble-based technique for approximately solving the partially observedproblem. Empirically, we demonstrate that our simple algorithm derived from theepistemic POMDP achieves significant gains in generalization over currentmethods on the Procgen benchmark suite.