Abstract
Guilt aversion induces experience of a utility loss in people if they believethey have disappointed others, and this promotes cooperative behaviour inhuman. In psychological game theory, guilt aversion necessitates modelling ofagents that have theory about what other agents think, also known as Theory ofMind (ToM). We aim to build a new kind of affective reinforcement learningagents, called Theory of Mind Agents with Guilt Aversion (ToMAGA), which areequipped with an ability to think about the wellbeing of others instead of justself-interest. To validate the agent design, we use a general-sum game known asStag Hunt as a test bed. As standard reinforcement learning agents could learnsuboptimal policies in social dilemmas like Stag Hunt, we propose to usebelief-based guilt aversion as a reward shaping mechanism. We show that ourbelief-based guilt averse agents can efficiently learn cooperative behavioursin Stag Hunt Games.