Brain age estimation based on magnetic resonance imaging (MRI) is an activeresearch area in early diagnosis of some neurodegenerative diseases (e.g.Alzheimer, Parkinson, Huntington, etc.) for elderly people or brainunderdevelopment for the young group. Deep learning methods have achieved thestate-of-the-art performance in many medical image analysis tasks, includingbrain age estimation. However, the performance and generalisability of the deeplearning model are highly dependent on the quantity and quality of the trainingdata set. Both collecting and annotating brain MRI data are extremelytime-consuming. In this paper, to overcome the data scarcity problem, wepropose a generative adversarial network (GAN) based image synthesis method.Different from the existing GAN-based methods, we integrate a task-guidedbranch (a regression model for age estimation) to the end of the generator inGAN. By adding a task-guided loss to the conventional GAN loss, the learnedlow-dimensional latent space and the synthesised images are more task-specific.It helps to boost the performance of the down-stream task by combining thesynthesised images and real images for model training. The proposed method wasevaluated on a public brain MRI data set for age estimation. Our proposedmethod outperformed (statistically significant) a deep convolutional neuralnetwork based regression model and the GAN-based image synthesis method withoutthe task-guided branch. More importantly, it enables the identification ofage-related brain regions in the image space. The code is available on GitHub(https://github.com/ruizhe-l/tgb-gan).