Abstract
Reward design for reinforcement learning agents can be difficult insituations where one not only wants the agent to achieve some effect in theworld but where one also cares about how that effect is achieved. For example,we might wish for an agent to adhere to a tacit understanding of commonsense,align itself to a preference for how to behave for purposes of safety, ortaking on a particular role in an interactive game. Storytelling is a mode forcommunicating tacit procedural knowledge. We introduce a technique, StoryShaping, in which a reinforcement learning agent infers tacit knowledge from anexemplar story of how to accomplish a task and intrinsically rewards itself forperforming actions that make its current environment adhere to that of theinferred story world. Specifically, Story Shaping infers a knowledge graphrepresentation of the world state from observations, and also infers aknowledge graph from the exemplar story. An intrinsic reward is generated basedon the similarity between the agent's inferred world state graph and theinferred story world graph. We conducted experiments in text-based gamesrequiring commonsense reasoning and shaping the behaviors of agents as virtualgame characters.