Abstract
Multi-agent reinforcement learning (MARL) methods struggle with thenon-stationarity of multi-agent systems and fail to adaptively learn onlinewhen tested with novel agents. Here, we leverage large language models (LLMs)to create an autonomous agent that can handle these challenges. Our agent,Hypothetical Minds, consists of a cognitively-inspired architecture, featuringmodular components for perception, memory, and hierarchical planning over twolevels of abstraction. We introduce the Theory of Mind module that scaffoldsthe high-level planning process by generating hypotheses about other agents'strategies in natural language. It then evaluates and iteratively refines thesehypotheses by reinforcing hypotheses that make correct predictions about theother agents' behavior. Hypothetical Minds significantly improves performanceover previous LLM-agent and RL baselines on a range of competitive, mixedmotive, and collaborative domains in the Melting Pot benchmark, including bothdyadic and population-based environments. Additionally, comparisons againstLLM-agent baselines and ablations reveal the importance of hypothesisevaluation and refinement for succeeding on complex scenarios.