Do deep reinforcement learning agents model intentions?

  • 2018-05-21 14:42:57
  • Tambet Matiisen, Aqeel Labash, Daniel Majoral, Jaan Aru, Raul Vicente
  • 0

Abstract

Inferring other agents' mental states such as their knowledge, beliefs andintentions is thought to be essential for effective interactions with otheragents. Recently, multiagent systems trained via deep reinforcement learninghave been shown to succeed in solving different tasks, but it remains unclearhow each agent modeled or represented other agents in their environment. Inthis work we test whether deep reinforcement learning agents explicitlyrepresent other agents' intentions (their specific aims or goals) during a taskin which the agents had to coordinate the covering of different spots in a 2Denvironment. In particular, we tracked over time the performance of a lineardecoder trained to predict the final goal of all agents from the hidden stateof each agent's neural network controller. We observed that the hidden layersof agents represented explicit information about other agents' goals, i.e. thetarget landmark they ended up covering. We also performed a series ofexperiments, in which some agents were replaced by others with fixed goals, totest the level of generalization of the trained agents. We noticed that duringthe training phase the agents developed a differential preference for eachgoal, which hindered generalization. To alleviate the above problem, we proposesimple changes to the MADDPG training algorithm which leads to bettergeneralization against unseen agents. We believe that training protocolspromoting more active intention reading mechanisms, e.g. by preventing simplesymmetry-breaking solutions, is a promising direction towards achieving a morerobust generalization in different cooperative and competitive tasks.

 

Quick Read (beta)

loading the full paper ...