Brain decoding, understood as the process of mapping brain activities to thestimuli that generated them, has been an active research area in the lastyears. In the case of language stimuli, recent studies have shown that it ispossible to decode fMRI scans into an embedding of the word a subject isreading. However, such word embeddings are designed for natural languageprocessing tasks rather than for brain decoding. Therefore, they limit ourability to recover the precise stimulus. In this work, we propose to directlyclassify an fMRI scan, mapping it to the corresponding word within a fixedvocabulary. Unlike existing work, we evaluate on scans from previously unseensubjects. We argue that this is a more realistic setup and we present a modelthat can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1and 13.59% Top-5 accuracy in this challenging task, significantly outperformingall the considered competitive baselines. Furthermore, we use the decoded wordsto guide language generation with the GPT-2 model. This way, we advance thequest for a system that translates brain activities into coherent text.