Abstract
Automatic human affect recognition is a key step towards more naturalhuman-computer interaction. Recent trends include recognition in the wild usinga fusion of audiovisual and physiological sensors, a challenging setting forconventional machine learning algorithms. Since 2010, novel deep learningalgorithms have been applied increasingly in this field. In this paper, wereview the literature on human affect recognition between 2010 and 2017, with aspecial focus on approaches using deep neural networks. By classifying a totalof 950 studies according to their usage of shallow or deep architectures, weare able to show a trend towards deep learning. Reviewing a subset of 233studies that employ deep neural networks, we comprehensively quantify theirapplications in this field. We find that deep learning is used for learning of(i) spatial feature representations, (ii) temporal feature representations, and(iii) joint feature representations for multimodal sensor data. Exemplarystate-of-the-art architectures illustrate the progress. Our findings show therole deep architectures will play in human affect recognition, and can serve asa reference point for researchers working on related applications.