Abstract
Zero-shot reinforcement learning (RL) has emerged as a setting for developinggeneral agents in an unsupervised manner, capable of solving downstream taskswithout additional training or planning at test-time. Unlike conventional RL,which optimizes policies for a fixed reward, zero-shot RL requires agents toencode representations rich enough to support immediate adaptation to anyobjective, drawing parallels to vision and language foundation models. Despitegrowing interest, the field lacks a common analytical lens. We present the first unified framework for zero-shot RL. Our formulationintroduces a consistent notation and taxonomy that organizes existingapproaches and allows direct comparison between them. Central to our frameworkis the classification of algorithms into two families: direct representations,which learn end-to-end mappings from rewards to policies, and compositionalrepresentations, which decompose the representation leveraging the substructureof the value function. Within this framework, we highlight shared principlesand key differences across methods, and we derive an extended bound forsuccessor-feature methods, offering a new perspective on their performance inthe zero-shot regime. By consolidating existing work under a common lens, ourframework provides a principled foundation for future research in zero-shot RLand outlines a clear path toward developing more general agents.