Abstract

It can be challenging to train multi-task neural networks that outperform oreven match their single-task counterparts. To help address this, we proposeusing knowledge distillation where single-task models teach a multi-task model.We enhance this training with teacher annealing, a novel method that graduallytransitions the model from distillation to supervised learning, helping themulti-task model surpass its single-task teachers. We evaluate our approach bymulti-task fine-tuning BERT on the GLUE benchmark. Our method consistentlyimproves over standard single-task and multi-task training.

 

Quick Read (beta)