Abstract
How related are the representations learned by neural language models,translation models, and language tagging tasks? We answer this question byadapting an encoder-decoder transfer learning method from computer vision toinvestigate the structure among 100 different feature spaces extracted fromhidden representations of various networks trained on language tasks. Thismethod reveals a low-dimensional structure where language models andtranslation models smoothly interpolate between word embeddings, syntactic andsemantic tasks, and future word embeddings. We call this low-dimensionalstructure a language representation embedding because it encodes therelationships between representations needed to process language for a varietyof NLP tasks. We find that this representation embedding can predict how welleach individual feature space maps to human brain responses to natural languagestimuli recorded using fMRI. Additionally, we find that the principal dimensionof this structure can be used to create a metric which highlights the brain'snatural language processing hierarchy. This suggests that the embeddingcaptures some part of the brain's natural language representation structure.