Abstract
Spatio-temporal localization is vital for precise interactions across diversedomains, from biological research to autonomous navigation and interactiveinterfaces. Current video-based approaches, while proficient in tracking, lackthe sophisticated reasoning capabilities of large language models, limitingtheir contextual understanding and generalization. We introduce VideoMolmo, alarge multimodal model tailored for fine-grained spatio-temporal pointingconditioned on textual descriptions. Building upon the Molmo architecture,VideoMolmo incorporates a temporal module utilizing an attention mechanism tocondition each frame on preceding frames, ensuring temporal consistency.Additionally, our novel temporal mask fusion pipeline employs SAM2 forbidirectional point propagation, significantly enhancing coherence across videosequences. This two-step decomposition, i.e., first using the LLM to generateprecise pointing coordinates, then relying on a sequential mask-fusion moduleto produce coherent segmentation, not only simplifies the task for the languagemodel but also enhances interpretability. Due to the lack of suitable datasets,we curate a comprehensive dataset comprising 72k video-caption pairs annotatedwith 100k object points. To evaluate the generalization of VideoMolmo, weintroduce VPoS-Bench, a challenging out-of-distribution benchmark spanning fivereal-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving,Video-GUI Interaction, and Robotics. We also evaluate our model on ReferringVideo Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison toexisting models, VideoMolmo substantially improves spatio-temporal pointingaccuracy and reasoning capability. Our code and models are publicly availableat https://github.com/mbzuai-oryx/VideoMolmo.