Learning segmentation from point trajectories

  • 2025-01-21 18:59:53
  • Laurynas Karazija, Iro Laina, Christian Rupprecht, Andrea Vedaldi
  • 0

Abstract

We consider the problem of segmenting objects in videos based on their motionand no other forms of supervision. Prior work has often approached this problemby using the principle of common fate, namely the fact that the motion ofpoints that belong to the same object is strongly correlated. However, mostauthors have only considered instantaneous motion from optical flow. In thiswork, we present a way to train a segmentation network using long-term pointtrajectories as a supervisory signal to complement optical flow. The keydifficulty is that long-term motion, unlike instantaneous motion, is difficultto model -- any parametric approximation is unlikely to capture complex motionpatterns over long periods of time. We instead draw inspiration from subspaceclustering approaches, proposing a loss function that seeks to group thetrajectories into low-rank matrices where the motion of object points can beapproximately explained as a linear combination of other point tracks. Ourmethod outperforms the prior art on motion-based segmentation, which shows theutility of long-term motion and the effectiveness of our formulation.

 

Quick Read (beta)

loading the full paper ...