A growing number of state-of-the-art transfer learning methods employlanguage models pretrained on large generic corpora. In this paper we present aconceptually simple and effective transfer learning approach that addresses theproblem of catastrophic forgetting. Specifically, we combine the task-specificoptimization function with an auxiliary language model objective, which isadjusted during the training process. This preserves language regularitiescaptured by language models, while enabling sufficient adaptation for solvingthe target task. Our method does not require pretraining or finetuning separatecomponents of the network and we train our models end-to-end in a single step.We present results on a variety of challenging affective and textclassification tasks, surpassing well established transfer learning methodswith greater level of complexity.