Decoding Continuous Character-based Language from Non-invasive Brain Recordings

  • 2024-03-19 11:09:20
  • Cenyuan Zhang, Xiaoqing Zheng, Ruicheng Yin, Shujie Geng, Jianhan Xu, Xuan Gao, Changze Lv, Zixuan Ling, Xuanjing Huang, Miao Cao, Jianfeng Feng
  • 0


Deciphering natural language from brain activity through non-invasive devicesremains a formidable challenge. Previous non-invasive decoders either requiremultiple experiments with identical stimuli to pinpoint cortical regions andenhance signal-to-noise ratios in brain activity, or they are limited todiscerning basic linguistic elements such as letters and words. We propose anovel approach to decoding continuous language from single-trial non-invasivefMRI recordings, in which a three-dimensional convolutional network augmentedwith information bottleneck is developed to automatically identify responsivevoxels to stimuli, and a character-based decoder is designed for the semanticreconstruction of continuous language characterized by inherent characterstructures. The resulting decoder can produce intelligible textual sequencesthat faithfully capture the meaning of perceived speech both within and acrosssubjects, while existing decoders exhibit significantly inferior performance incross-subject contexts. The ability to decode continuous language from singletrials across subjects demonstrates the promising applications of non-invasivelanguage brain-computer interfaces in both healthcare and neuroscience.


Quick Read (beta)

loading the full paper ...