Abstract
Reinforcement learning (RL) algorithms are highly sensitive to rewardfunction specification, which remains a central challenge limiting their broadapplicability. We present ARM-FM: Automated Reward Machines via FoundationModels, a framework for automated, compositional reward design in RL thatleverages the high-level reasoning capabilities of foundation models (FMs).Reward machines (RMs) -- an automata-based formalism for reward specification-- are used as the mechanism for RL objective specification, and areautomatically constructed via the use of FMs. The structured formalism of RMsyields effective task decompositions, while the use of FMs enables objectivespecifications in natural language. Concretely, we (i) use FMs to automaticallygenerate RMs from natural language specifications; (ii) associate languageembeddings with each RM automata-state to enable generalization across tasks;and (iii) provide empirical evidence of ARM-FM's effectiveness in a diversesuite of challenging environments, including evidence of zero-shotgeneralization.