One third of stroke survivors have language difficulties. Emerging evidencesuggests that their likelihood of recovery depends mainly on the damage tolanguage centers. Thus previous research for predicting language recoverypost-stroke has focused on identifying damaged regions of the brain. In thispaper, we introduce a novel method where we only make use of stitched2-dimensional cross-sections of raw MRI scans in a deep convolutional neuralnetwork setup to predict language recovery post-stroke. Our results show: a)the proposed model that only uses MRI scans has comparable performance tomodels that are dependent on lesion specific information; b) the featureslearned by our model are complementary to the lesion specific information andthe combination of both appear to outperform previously reported results insimilar settings. We further analyse the CNN model for understanding regions inbrain that are responsible for arriving at these predictions using gradientbased saliency maps. Our findings are in line with previous lesion studies.