Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

  • 2022-09-28 18:36:47
  • Lukas Christ, Shahin Amiriparian, Alexander Kathan, Niklas Müller, Andreas König, Björn W. Schuller
  • 2

Abstract

Humour is a substantial element of human affect and cognition. Its automaticunderstanding can facilitate a more naturalistic human-device interaction andthe humanisation of artificial intelligence. Current methods of humourdetection are solely based on staged data making them inadequate for'real-world' applications. We address this deficiency by introducing the novelPassau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising ofabout 11 hours of recordings. The Passau-SFCH dataset is annotated for thepresence of humour and its dimensions (sentiment and direction) as proposed inMartin's Humor Style Questionnaire. We conduct a series of experiments,employing pretrained Transformers, convolutional neural networks, andexpert-designed features. The performance of each modality (text, audio, video)for spontaneous humour recognition is analysed and their complementarity isinvestigated. Our findings suggest that for the automatic analysis of humourand its sentiment, facial expressions are most promising, while humourdirection can be best modelled via text-based features. The results revealconsiderable differences among various subjects, highlighting the individualityof humour usage and style. Further, we observe that a decision-level fusionyields the best recognition result. Finally, we make our code publiclyavailable at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH datasetis available upon request.

 

Quick Read (beta)

loading the full paper ...