Abstract
Generating human language through non-invasive brain-computer interfaces(BCIs) has the potential to unlock many applications, such as serving disabledpatients and improving communication. Currently, however, generating languagevia BCIs has been previously successful only within a classification setup forselecting pre-generated sentence continuation candidates with the most likelycortical semantic representation. Inspired by recent research that revealedassociations between the brain and the large computational language models, wepropose a generative language BCI that utilizes the capacity of a largelanguage model (LLM) jointly with a semantic brain decoder to directly generatelanguage from functional magnetic resonance imaging (fMRI) input. The proposedmodel can generate coherent language sequences aligned with the semanticcontent of visual or auditory language stimuli perceived, without priorknowledge of any pre-generated candidates. We compare the language generatedfrom the presented model with a random control, pre-generated languageselection approach, and a standard LLM, which generates common coherent textsolely based on the next word likelihood according to statistical languagetraining data. The proposed model is found to generate language that is morealigned with semantic stimulus in response to which brain input is sampled. Ourfindings demonstrate the potential and feasibility of employing BCIs in directlanguage generation.