Abstract
Pre-trained neural language models bring significant improvement for variousNLP tasks, by fine-tuning the models on task-specific training sets. Duringfine-tuning, the parameters are initialized from pre-trained models directly,which ignores how the learning process of similar NLP tasks in differentdomains is correlated and mutually reinforced. In this paper, we propose aneffective learning procedure named Meta Fine-Tuning (MFT), served as ameta-learner to solve a group of similar NLP tasks for neural language models.Instead of simply multi-task training over all the datasets, MFT only learnsfrom typical instances of various domains to acquire highly transferableknowledge. It further encourages the language model to encode domain-invariantrepresentations by optimizing a series of novel domain corruption lossfunctions. After MFT, the model can be fine-tuned for each domain with betterparameter initializations and higher generalization ability. We implement MFTupon BERT to solve several multi-domain text mining tasks. Experimental resultsconfirm the effectiveness of MFT and its usefulness for few-shot learning.