Abstract
Learning from demonstration (LfD) is useful in settings where hand-codingbehaviour or a reward function is impractical. It has succeeded in a wide rangeof problems but typically relies on artificially generated demonstrations orspecially deployed sensors and has not generally been able to leverage thecopious demonstrations available in the wild: those that capture behaviour thatwas occurring anyway using sensors that were already deployed for anotherpurpose, e.g., traffic camera footage capturing demonstrations of naturalbehaviour of vehicles, cyclists, and pedestrians. We propose video to behaviour(ViBe), a new approach to learning models of road user behaviour that requiresas input only unlabelled raw video data of a traffic scene collected from asingle, monocular, uncalibrated camera with ordinary resolution. Our approachcalibrates the camera, detects relevant objects, tracks them through time, anduses the resulting trajectories to perform LfD, yielding models of naturalisticbehaviour. We apply ViBe to raw videos of a traffic intersection and show thatit can learn purely from videos, without additional expert knowledge.