Abstract
We propose an unsupervised learning approach using a convolutional and fullyconnected autoencoder, which we call deep heterogeneous autoencoder, to learndiscriminative features from segmentation masks and detection bounding boxes.To learn the mask shape information and its corresponding location in an inputimage, we extract coarse masks from a pretrained semantic segmentation networkas well as their corresponding bounding boxes. We train the autoencodersjointly using task-dependent uncertainty weights to generate common latentfeatures. The feature vector is then fed to the k-means clustering algorithm toseparate the data points in the latent space. Finally, we incorporateadditional penalties in the form of a constraints graph based on priorknowledge of the sequential data to increase clustering robustness. We evaluatethe performance of our method using both synthetic and real world multi-objectvideo datasets to demonstrate the applicability of our proposed model. Ourresults show that the proposed technique outperforms several state-of-the-artmethods on challenging video sequences.