Abstract
We present ALFRED (Action Learning From Realistic Environments andDirectives), a benchmark for learning a mapping from natural languageinstructions and egocentric vision to sequences of actions for household tasks.ALFRED includes long, compositional tasks with non-reversible state changes toshrink the gap between research benchmarks and real-world applications. ALFREDconsists of expert demonstrations in interactive visual environments for 25knatural language directives. These directives contain both high-level goalslike "Rinse off a mug and place it in the coffee maker." and low-level languageinstructions like "Walk to the coffee maker on the right." ALFRED tasks aremore complex in terms of sequence length, action space, and language thanexisting vision-and-language task datasets. We show that a baseline model basedon recent embodied vision-and-language tasks performs poorly on ALFRED,suggesting that there is significant room for developing innovative groundedvisual language understanding models with this benchmark.