Abstract
Sometimes we benefit from actions that others have taken even when we areunaware that they took those actions. For example, if your neighbor chooses notto take a parking spot in front of your house when you are not there, you canbenefit, even without being aware that they took this action. These ``hiddengifts'' represent an interesting challenge for multi-agent reinforcementlearning (MARL), since assigning credit when the beneficial actions of othersare hidden is non-trivial. Here, we study the impact of hidden gifts with avery simple MARL task. In this task, agents in a grid-world environment haveindividual doors to unlock in order to obtain individual rewards. As well, ifall the agents unlock their door the group receives a larger collective reward.However, there is only one key for all of the doors, such that the collectivereward can only be obtained when the agents drop the key for others after theyuse it. Notably, there is nothing to indicate to an agent that the other agentshave dropped the key, thus this act for others is a ``hidden gift''. We showthat several different state-of-the-art MARL algorithms, including MARLspecific architectures, fail to learn how to obtain the collective reward inthis simple task. Interestingly, we find that decentralized actor-critic policygradient agents can succeed when we provide them with information about theirown action history, but MARL agents still cannot solve the task with actionhistory. Finally, we derive a correction term for policy gradient agents,inspired by learning aware approaches, which reduces the variance in learningand helps them to converge to collective success more reliably. These resultsshow that credit assignment in multi-agent settings can be particularlychallenging in the presence of ``hidden gifts'', and demonstrate that selflearning-awareness in decentralized agents can benefit these settings.