A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

  • 2022-07-23 00:19:57
  • Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer
  • 0


Vision-language navigation (VLN), in which an agent follows languageinstruction in a visual environment, has been studied under the premise thatthe input command is fully feasible in the environment. Yet in practice, arequest may not be possible due to language ambiguity or environment changes.To study VLN with unknown command feasibility, we introduce a new datasetMobile app Tasks with Iterative Feedback (MoTIF), where the goal is to completea natural language command in a mobile app. Mobile apps provide a scalabledomain to study real downstream uses of VLN methods. Moreover, mobile appcommands provide instruction for interactive navigation, as they result inaction sequences with state changes via clicking, typing, or swiping. MoTIF isthe first to include feasibility annotations, containing both binaryfeasibility labels and fine-grained labels for why tasks are unsatisfiable. Wefurther collect follow-up questions for ambiguous queries to enable research ontask uncertainty resolution. Equipped with our dataset, we propose the newproblem of feasibility prediction, in which a natural language instruction andmultimodal app environment are used to predict command feasibility. MoTIFprovides a more realistic app dataset as it contains many diverse environments,high-level goals, and longer action sequences than prior work. We evaluateinteractive VLN methods using MoTIF, quantify the generalization ability ofcurrent approaches to new app environments, and measure the effect of taskfeasibility on navigation performance.


Quick Read (beta)

loading the full paper ...