Abstract
Multimodal language modeling constitutes a recent breakthrough whichleverages advances in large language models to pretrain capable multimodalmodels. The integration of natural language during pretraining has been shownto significantly improve learned representations, particularly in computervision. However, the efficacy of multimodal language modeling in the realm offunctional brain data, specifically for advancing pathology detection, remainsunexplored. This study pioneers EEG-language models trained on clinical reportsand 15000 EEGs. We extend methods for multimodal alignment to this novel domainand investigate which textual information in reports is useful for trainingEEG-language models. Our results indicate that models learn richerrepresentations from being exposed to a variety of report segments, includingthe patient's clinical history, description of the EEG, and the physician'sinterpretation. Compared to models exposed to narrower clinical textinformation, we find such models to retrieve EEGs based on clinical reports(and vice versa) with substantially higher accuracy. Yet, this is only observedwhen using a contrastive learning approach. Particularly in regimes with fewannotations, we observe that representations of EEG-language models cansignificantly improve pathology detection compared to those of EEG-only models,as demonstrated by both zero-shot classification and linear probes. In sum,these results highlight the potential of integrating brain activity data withclinical text, suggesting that EEG-language models represent significantprogress for clinical applications.