In recent years, transformer models have achieved great success in naturallanguage processing (NLP) tasks. Most of the current state-of-the-art NLPresults are achieved by using monolingual transformer models, where the modelis pre-trained using a single language unlabelled text corpus. Then, the modelis fine-tuned to the specific downstream task. However, the cost ofpre-training a new transformer model is high for most languages. In this work,we propose a cost-effective transfer learning method to adopt a strong sourcelanguage model, trained from a large monolingual corpus to a low-resourcelanguage. Thus, using XLNet language model, we demonstrate competitiveperformance with mBERT and a pre-trained target language model on thecross-lingual sentiment (CLS) dataset and on a new sentiment analysis datasetfor low-resourced language Tigrinya. With only 10k examples of the givenTigrinya sentiment analysis dataset, English XLNet has achieved 78.88% F1-Scoreoutperforming BERT and mBERT by 10% and 7%, respectively. More interestingly,fine-tuning (English) XLNet model on the CLS dataset has promising resultscompared to mBERT and even outperformed mBERT for one dataset of the Japaneselanguage.