Abstract
Self-supervised learning has drawn attention through its effectiveness inlearning in-domain representations with no ground-truth annotations; inparticular, it is shown that properly designed pretext tasks (e.g., contrastiveprediction task) bring significant performance gains for downstream tasks(e.g., classification task). Inspired from this, we tackle video scenesegmentation, which is a task of temporally localizing scene boundaries in avideo, with a self-supervised learning framework where we mainly focus ondesigning effective pretext tasks. In our framework, we discover apseudo-boundary from a sequence of shots by splitting it into two continuous,non-overlapping sub-sequences and leverage the pseudo-boundary to facilitatethe pre-training. Based on this, we introduce three novel boundary-awarepretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM)and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximizeintra-scene similarity and inter-scene discrimination while PP encourages themodel to identify transitional moments. Through comprehensive analysis, weempirically show that pre-training and transferring contextual representationare both critical to improving the video scene segmentation performance.Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. Thecode is available at https://github.com/kakaobrain/bassl.