Abstract
Deciphering natural language from brain activity through non-invasive devicesremains a formidable challenge. Previous non-invasive decoders either requiremultiple experiments with identical stimuli to pinpoint cortical regions andenhance signal-to-noise ratios in brain activity, or they are limited todiscerning basic linguistic elements such as letters and words. We propose anovel approach to decoding continuous language from single-trial non-invasivefMRI recordings, in which a three-dimensional convolutional network augmentedwith information bottleneck is developed to automatically identify responsivevoxels to stimuli, and a character-based decoder is designed for the semanticreconstruction of continuous language characterized by inherent characterstructures. The resulting decoder can produce intelligible textual sequencesthat faithfully capture the meaning of perceived speech both within and acrosssubjects, while existing decoders exhibit significantly inferior performance incross-subject contexts. The ability to decode continuous language from singletrials across subjects demonstrates the promising applications of non-invasivelanguage brain-computer interfaces in both healthcare and neuroscience.