Abstract
The ability to adapt to physical actions and constraints in an environment iscrucial for embodied agents (e.g., robots) to effectively collaborate withhumans. Such physically grounded human-AI collaboration must account for theincreased complexity of the continuous state-action space and constraineddynamics caused by physical constraints. In this paper, we introduce\textit{Moving Out}, a new human-AI collaboration benchmark that resembles awide range of collaboration modes affected by physical attributes andconstraints, such as moving heavy items together and maintaining consistentactions to move a big item around a corner. Using Moving Out, we designed twotasks and collected human-human interaction data to evaluate models' abilitiesto adapt to diverse human behaviors and unseen physical attributes. To addressthe challenges in physical environments, we propose a novel method, BASS(Behavior Augmentation, Simulation, and Selection), to enhance the diversity ofagents and their understanding of the outcome of actions. Our experiments showthat BASS outperforms state-of-the-art models in AI-AI and human-AIcollaboration. The project page is available at\href{https://live-robotics-uva.github.io/movingout_ai/}{https://live-robotics-uva.github.io/movingout\_ai/}.