Abstract
In cooperative multiagent reinforcement learning, agents commonly acquireheterogeneous knowledge. Learning across the team can be greatly improved ifagents can effectively exchange their knowledge to other agents. In particular,recent work showed that action advising, a form of peer-to-peer knowledgetransfer from teacher agents to student agents, improves team-wide learning.However, that prior work on action advising only considered advising withprimitive (low-level) actions, which limits scalability. This paper introducesa novel learning-to-teach framework, called hierarchical multiagent teaching(HMAT), in which the teacher advice may include extended action sequences overmultiple levels of temporal abstraction. The empirical evaluations show thatHMAT accelerates team-wide learning progress in environments that are morecomplex than considered in previous learning-to-teach research. HMAT is alsoshown to learn teaching policies that can be transferred to differentteammates/tasks, even when teammates have heterogeneous action spaces.