Due to the high availability of large-scale annotated image datasets,knowledge transfer from pre-trained models showed outstanding performance inmedical image classification. However, building a robust image classificationmodel for datasets with data irregularity or imbalanced classes can be a verychallenging task, especially in the medical imaging domain. In this paper, wepropose a novel deep convolutional neural network, we called Self SupervisedSuper Sample Decomposition for Transfer learning (4S-DT) model. 4S-DTencourages a coarse-to-fine transfer learning from large-scale imagerecognition tasks to a specific chest X-ray image classification task using ageneric self-supervised sample decomposition approach. Our main contribution isa novel self-supervised learning mechanism guided by a super sampledecomposition of unlabelled chest X-ray images. 4S-DT helps in improving therobustness of knowledge transformation via a downstream learning strategy witha class-decomposition layer to simplify the local structure of the data. 4S-DTcan deal with any irregularities in the image dataset by investigating itsclass boundaries using a downstream class-decomposition mechanism. We used50,000 unlabelled chest X-ray images to achieve our coarse-to-fine transferlearning with an application to COVID-19 detection, as an exemplar. 4S-DT hasachieved a high accuracy of 99.8% (95% CI: 99.44%, 99.98%) in the detection ofCOVID-19 cases on a large dataset and an accuracy of 97.54% (95%$ CI: 96.22%,98.91%) on an extended test set enriched by augmented images of a smalldataset, out of which all real COVID-19 cases were detected, which was thehighest accuracy obtained when compared to other methods.