Abstract
Music is a universal phenomenon that profoundly influences human experiencesacross cultures. This study investigates whether music can be decoded fromhuman brain activity measured with functional MRI (fMRI) during its perception.Leveraging recent advancements in extensive datasets and pre-trainedcomputational models, we construct mappings between neural data and latentrepresentations of musical stimuli. Our approach integrates functional andanatomical alignment techniques to facilitate cross-subject decoding,addressing the challenges posed by the low temporal resolution andsignal-to-noise ratio (SNR) in fMRI data. Starting from the GTZan fMRI dataset,where five participants listened to 540 musical stimuli from 10 differentgenres while their brain activity was recorded, we used the CLAP (ContrastiveLanguage-Audio Pretraining) model to extract latent representations of themusical stimuli and developed voxel-wise encoding models to identify brainregions responsive to these stimuli. By applying a threshold to the associationbetween predicted and actual brain activity, we identified specific regions ofinterest (ROIs) which can be interpreted as key players in music processing.Our decoding pipeline, primarily retrieval-based, employs a linear map toproject brain activity to the corresponding CLAP features. This enables us topredict and retrieve the musical stimuli most similar to those that originatedthe fMRI data. Our results demonstrate state-of-the-art identificationaccuracy, with our methods significantly outperforming existing approaches. Ourfindings suggest that neural-based music retrieval systems could enablepersonalized recommendations and therapeutic applications. Future work coulduse higher temporal resolution neuroimaging and generative models to improvedecoding accuracy and explore the neural underpinnings of music perception andemotion.