Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

  • 2019-10-08 16:08:40
  • Khanh Nguyen, Hal Daumé III
  • 0


Mobile agents that can leverage help from humans can potentially accomplishmore complex tasks than they could entirely on their own. We develop "Help,Anna!" (HANNA), an interactive photo-realistic simulator in which an agentfulfills object-finding tasks by requesting and interpreting naturallanguage-and-vision assistance. An agent solving tasks in a HANNA environmentcan leverage simulated human assistants, called ANNA (Automatic NaturalNavigation Assistants), which, upon request, provide natural language andvisual instructions to direct the agent towards the goals. To address the HANNAproblem, we develop a memory-augmented neural agent that hierarchically modelsmultiple levels of decision-making, and an imitation learning algorithm thatteaches the agent to avoid repeating past mistakes while simultaneouslypredicting its own chances of making future progress. Empirically, our approachis able to ask for help more effectively than competitive baselines and, thus,attains higher task success rate on both previously seen and previously unseenenvironments. We publicly release code and data at .


Quick Read (beta)

This feature is not avaialbe for this paper.