Abstract
Recent advances in multi-agent reinforcement learning (MARL) have createdopportunities to solve complex real-world tasks. Cybersecurity is a notableapplication area, where defending networks against sophisticated adversariesremains a challenging task typically performed by teams of security operators.In this work, we explore novel MARL strategies for building autonomous cybernetwork defenses that address challenges such as large policy spaces, partialobservability, and stealthy, deceptive adversarial strategies. To facilitateefficient and generalized learning, we propose a hierarchical Proximal PolicyOptimization (PPO) architecture that decomposes the cyber defense task intospecific sub-tasks like network investigation and host recovery. Our approachinvolves training sub-policies for each sub-task using PPO enhanced with domainexpertise. These sub-policies are then leveraged by a master defense policythat coordinates their selection to solve complex network defense tasks.Furthermore, the sub-policies can be fine-tuned and transferred with minimalcost to defend against shifts in adversarial behavior or changes in networksettings. We conduct extensive experiments using CybORG Cage 4, thestate-of-the-art MARL environment for cyber defense. Comparisons with multiplebaselines across different adversaries show that our hierarchical learningapproach achieves top performance in terms of convergence speed, episodicreturn, and several interpretable metrics relevant to cybersecurity, includingthe fraction of clean machines on the network, precision, and false positiveson recoveries.