SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigation

  • 2025-06-17 06:47:59
  • Xiangyu Shi, Zerui Li, Wenqi Lyu, Jiatong Xia, Feras Dayoub, Yanyuan Qiao, Qi Wu
  • 0

Abstract

Vision-and-Language Navigation (VLN) in continuous environments requiresagents to interpret natural language instructions while navigatingunconstrained 3D spaces. Existing VLN-CE frameworks rely on a two-stageapproach: a waypoint predictor to generate waypoints and a navigator to executemovements. However, current waypoint predictors struggle with spatialawareness, while navigators lack historical reasoning and backtrackingcapabilities, limiting adaptability. We propose a zero-shot VLN-CE frameworkintegrating an enhanced waypoint predictor with a Multi-modal Large LanguageModel (MLLM)-based navigator. Our predictor employs a stronger vision encoder,masked cross-attention fusion, and an occupancy-aware loss for better waypointquality. The navigator incorporates history-aware reasoning and adaptive pathplanning with backtracking, improving robustness. Experiments on R2R-CE andMP3D benchmarks show our method achieves state-of-the-art (SOTA) performance inzero-shot settings, demonstrating competitive results compared to fullysupervised methods. Real-world validation on Turtlebot 4 further highlights itsadaptability.

 

Quick Read (beta)

loading the full paper ...