Abstract
While reinforcement learning from scratch has shown impressive results insolving sequential decision-making tasks with efficient simulators, real-worldapplications with expensive interactions require more sample-efficient agents.Foundation models (FMs) are natural candidates to improve sample efficiency asthey possess broad knowledge and reasoning capabilities, but it is yet unclearhow to effectively integrate them into the reinforcement learning framework. Inthis paper, we anticipate and, most importantly, evaluate two promisingstrategies. First, we consider the use of foundation world models (FWMs) thatexploit the prior knowledge of FMs to enable training and evaluating agentswith simulated interactions. Second, we consider the use of foundation agents(FAs) that exploit the reasoning capabilities of FMs for decision-making. Weevaluate both approaches empirically in a family of grid-world environmentsthat are suitable for the current generation of large language models (LLMs).Our results suggest that improvements in LLMs already translate into betterFWMs and FAs; that FAs based on current LLMs can already provide excellentpolicies for sufficiently simple environments; and that the coupling of FWMsand reinforcement learning agents is highly promising for more complex settingswith partial observability and stochastic elements.