Abstract
We study a collaborative scenario where a user not only instructs a system tocomplete tasks, but also acts alongside it. This allows the user to adapt tothe system abilities by changing their language or deciding to simplyaccomplish some tasks themselves, and requires the system to effectivelyrecover from errors as the user strategically assigns it new goals. We build agame environment to study this scenario, and learn to map user instructions tosystem actions. We introduce a learning approach focused on recovery fromcascading errors between instructions, and modeling methods to explicitlyreason about instructions with multiple goals. We evaluate with a newevaluation protocol using recorded interactions and online games with humanusers, and observe how users adapt to the system abilities.