Abstract
This paper presents BELT, a novel model and learning framework for thepivotal topic of brain-to-language translation research. The translation fromnoninvasive brain signals into readable natural language has the potential topromote the application scenario as well as the development of brain-computerinterfaces (BCI) as a whole. The critical problem in brain signal decoding orbrain-to-language translation is the acquisition of semantically appropriateand discriminative EEG representation from a dataset of limited scale andquality. The proposed BELT method is a generic and efficient framework thatbootstraps EEG representation learning using off-the-shelf large-scalepretrained language models (LMs). With a large LM's capacity for understandingsemantic information and zero-shot generalization, BELT utilizes large LMstrained on Internet-scale datasets to bring significant improvements to theunderstanding of EEG signals. In particular, the BELT model is composed of a deep conformer encoder and avector quantization encoder. Semantical EEG representation is achieved by acontrastive learning step that provides natural language supervision. Weachieve state-of-the-art results on two featuring brain decoding tasksincluding the brain-to-language translation and zero-shot sentimentclassification. Specifically, our model surpasses the baseline model on bothtasks by 5.45% and over 10% and archives a 42.31% BLEU-1 score and 67.32%precision on the main evaluation metrics for translation and zero-shotsentiment classification respectively.