Abstract
In shared autonomy, user input is combined with semi-autonomous control toachieve a common goal. The goal is often unknown ex-ante, so prior work enablesagents to infer the goal from user input and assist with the task. Such methodstend to assume some combination of knowledge of the dynamics of theenvironment, the user's policy given their goal, and the set of possible goalsthe user might target, which limits their application to real-world scenarios.We propose a deep reinforcement learning framework for model-free sharedautonomy that lifts these assumptions. We use human-in-the-loop reinforcementlearning with neural network function approximation to learn an end-to-endmapping from environmental observation and user input to agent action values,with task reward as the only form of supervision. This approach poses thechallenge of following user commands closely enough to provide the user withreal-time action feedback and thereby ensure high-quality user input, but alsodeviating from the user's actions when they are suboptimal. We balance thesetwo needs by discarding actions whose values fall below some threshold, thenselecting the remaining action closest to the user's input. Controlled studieswith users (n = 12) and synthetic pilots playing a video game, and a pilotstudy with users (n = 4) flying a real quadrotor, demonstrate the ability ofour algorithm to assist users with real-time control tasks in which the agentcannot directly access the user's private information through observations, butreceives a reward signal and user input that both depend on the user's intent.The agent learns to assist the user without access to this private information,implicitly inferring it from the user's input. This paper is a proof of conceptthat illustrates the potential for deep reinforcement learning to enableflexible and practical assistive systems.