Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees

  • 2023-05-20 22:13:00
  • Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, Matthew Gombolay
  • 0

Abstract

Human-AI policy specification is a novel procedure we define in which humanscan collaboratively warm-start a robot's reinforcement learning policy. Thisprocedure is comprised of two steps; (1) Policy Specification, i.e. humansspecifying the behavior they would like their companion robot to accomplish,and (2) Policy Optimization, i.e. the robot applying reinforcement learning toimprove the initial policy. Existing approaches to enabling collaborativepolicy specification are often unintelligible black-box methods, and are notcatered towards making the autonomous system accessible to a novice end-user.In this paper, we develop a novel collaborative framework to allow humans toinitialize and interpret an autonomous agent's behavior. Through our framework,we enable humans to specify an initial behavior model via unstructured, naturallanguage (NL), which we convert to lexical decision trees. Next, we leveragethese translated specifications, to warm-start reinforcement learning and allowthe agent to further optimize these potentially suboptimal policies. Ourapproach warm-starts an RL agent by utilizing non-expert natural languagespecifications without incurring the additional domain exploration costs. Wevalidate our approach by showing that our model is able to produce >80%translation accuracy, and that policies initialized by a human can match theperformance of relevant RL baselines in two domains.

 

Quick Read (beta)

loading the full paper ...