Abstract
Electroencephalography (EEG) and language have been widely exploredindependently for many downstream tasks (e.g., sentiment analysis, relationdetection, etc.). Multimodal approaches that study both domains have not beenwell explored, even though in recent years, multimodal learning has been seento be more powerful than its unimodal counterparts. In this study, we want toexplore the relationship and dependency between EEG and language, i.e., how onedomain reflects and represents the other. To study the relationship at therepresentation level, we introduced MTAM, a MultimodalTransformer AlignmentModel, to observe coordinated representations between the two modalities, andthus employ the transformed representations for downstream applications. Weused various relationship alignment-seeking techniques, such as CanonicalCorrelation Analysis and Wasserstein Distance, as loss functions to transfigurelow-level language and EEG features to high-level transformed features. Ondownstream applications, sentiment analysis and relation detection, we achievednew state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our methodachieved an F1-score improvement of 16.5% on sentiment analysis for K-EmoCon,27% on sentiment analysis of ZuCo, and 31.1% on relation detection of ZuCo. Inaddition, we provide interpretations of the performance improvement by: (1)visualizing the original feature distribution and the transformed featuredistribution, showing the effectiveness of the alignment module for discoveringand encoding the relationship between EEG and language; (2) visualizingword-level and sentence-level EEG-language alignment weights, showing theinfluence of different language semantics as well as EEG frequency features;and (3) visualizing brain topographical maps to provide an intuitivedemonstration of the connectivity of EEG and language response in the brainregions.