Abstract
In order for agents trained by deep reinforcement learning to work alongsidehumans in realistic settings, we will need to ensure that the agents are\emph{robust}. Since the real world is very diverse, and human behavior oftenchanges in response to agent deployment, the agent will likely encounter novelsituations that have never been seen during training. This results in anevaluation challenge: if we cannot rely on the average training or validationreward as a metric, then how can we effectively evaluate robustness? We takeinspiration from the practice of \emph{unit testing} in software engineering.Specifically, we suggest that when designing AI agents that collaborate withhumans, designers should search for potential edge cases in \emph{possiblepartner behavior} and \emph{possible states encountered}, and write tests whichcheck that the behavior of the agent in these edge cases is reasonable. Weapply this methodology to build a suite of unit tests for the Overcooked-AIenvironment, and use this test suite to evaluate three proposals for improvingrobustness. We find that the test suite provides significant insight into theeffects of these proposals that were generally not revealed by looking solelyat the average validation reward.