Abstract
Despite the popularity of multi-agent reinforcement learning (RL) insimulated and two-player applications, its success in messy real-worldapplications has been limited. A key challenge lies in its generalizabilityacross problem variations, a common necessity for many real-world problems.Contextual reinforcement learning (CRL) formalizes learning policies thatgeneralize across problem variations. However, the lack of standardizedbenchmarks for multi-agent CRL has hindered progress in the field. Suchbenchmarks are desired to be based on real-world applications to naturallycapture the many open challenges of real-world problems that affectgeneralization. To bridge this gap, we propose IntersectionZoo, a comprehensivebenchmark suite for multi-agent CRL through the real-world application ofcooperative eco-driving in urban road networks. The task of cooperativeeco-driving is to control a fleet of vehicles to reduce fleet-level vehicularemissions. By grounding IntersectionZoo in a real-world application, wenaturally capture real-world problem characteristics, such as partialobservability and multiple competing objectives. IntersectionZoo is built ondata-informed simulations of 16,334 signalized intersections derived from 10major US cities, modeled in an open-source industry-grade microscopic trafficsimulator. By modeling factors affecting vehicular exhaust emissions (e.g.,temperature, road conditions, travel demand), IntersectionZoo provides onemillion data-driven traffic scenarios. Using these traffic scenarios, webenchmark popular multi-agent RL and human-like driving algorithms anddemonstrate that the popular multi-agent RL algorithms struggle to generalizein CRL settings.