Abstract
Human emotions are complex and can be conveyed through nuanced touchgestures. Previous research has primarily focused on how humans recognizeemotions through touch or on identifying key features of emotional expressionfor robots. However, there is a gap in understanding how reliably theseemotions and gestures can be communicated to robots via touch and interpretedusing data driven methods. This study investigates the consistency anddistinguishability of emotional and gestural expressions through touch andsound. To this end, we integrated a custom piezoresistive pressure sensor aswell as a microphone on a social robot. Twenty-eight participants firstconveyed ten different emotions to the robot using spontaneous touch gestures,then they performed six predefined social touch gestures. Our findings revealstatistically significant consistency in both emotion and gesture expressionamong participants. However, some emotions exhibited low intraclass correlationvalues, and certain emotions with similar levels of arousal or valence did notshow significant differences in their conveyance. To investigate emotion andsocial gesture decoding within affective human-robot tactile interaction, wedeveloped single-modality models and multimodal models integrating tactile andauditory features. A support vector machine (SVM) model trained on multimodalfeatures achieved the highest accuracy for classifying ten emotions, reaching40 %.For gesture classification, a Convolutional Neural Network- LongShort-Term Memory Network (CNN-LSTM) achieved 90.74 % accuracy. Our resultsdemonstrate that even though the unimodal models have the potential to decodeemotions and touch gestures, the multimodal integration of touch and soundsignificantly outperforms unimodal approaches, enhancing the decoding of bothemotions and gestures.