Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

  • 2020-01-31 10:46:59
  • Milad Mozafari, Leila Reddy, Rufin VanRullen
  • 35

Abstract

Decoding and reconstructing images from brain imaging data is a research areaof high interest. Recent progress in deep generative neural networks hasintroduced new opportunities to tackle this problem. Here, we employ a recentlyproposed large-scale bi-directional generative adversarial network, calledBigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGANconverts images into a 120-dimensional latent space which encodes class andattribute information together, and can also reconstruct images based on theirlatent vectors. We trained a linear mapping between fMRI data, acquired overimages from 150 different categories of ImageNet, and their correspondingBigBiGAN latent vectors. Then, we applied this mapping to the fMRI activitypatterns obtained from 50 new test images from 50 unseen categories in order toretrieve their latent vectors, and reconstruct the corresponding images.Pairwise image decoding from the predicted latent vectors was highly accurate(84%). Moreover, qualitative and quantitative assessments revealed that theresulting image reconstructions were visually plausible, successfully capturedmany attributes of the original images, and had high perceptual similarity withthe original content. This method establishes a new state-of-the-art forfMRI-based natural image reconstruction, and can be flexibly updated to takeinto account any future improvements in generative models of natural sceneimages.

 

Quick Read (beta)

loading the full paper ...