Abstract
The emerging zero-shot capabilities of Large Language Models (LLMs) have ledto their applications in areas extending well beyond natural languageprocessing tasks. In reinforcement learning, while LLMs have been extensivelyused in text-based environments, their integration with continuous state spacesremains understudied. In this paper, we investigate how pre-trained LLMs can beleveraged to predict in context the dynamics of continuous Markov decisionprocesses. We identify handling multivariate data and incorporating the controlsignal as key challenges that limit the potential of LLMs' deployment in thissetup and propose Disentangled In-Context Learning (DICL) to address them. Wepresent proof-of-concept applications in two reinforcement learning settings:model-based policy evaluation and data-augmented off-policy reinforcementlearning, supported by theoretical analysis of the proposed methods. Ourexperiments further demonstrate that our approach produces well-calibrateduncertainty estimates. We release the code athttps://github.com/abenechehab/dicl.