Abstract
Adversarial training, which minimizes the maximal risk for label-preservinginput perturbations, has proved to be effective for improving thegeneralization of language models. In this work, we propose a novel adversarialtraining algorithm, FreeLB, that promotes higher invariance in the embeddingspace, by adding adversarial perturbations to word embeddings and minimizingthe resultant adversarial risk inside different regions around input samples.To validate the effectiveness of the proposed approach, we apply it toTransformer-based models for natural language understanding and commonsensereasoning tasks. Experiments on the GLUE benchmark show that when applied onlyto the finetuning stage, it is able to improve the overall test scores ofBERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8.In addition, the proposed approach achieves state-of-the-art single-model testaccuracies of 85.44\% and 67.75\% on ARC-Easy and ARC-Challenge. Experiments onCommonsenseQA benchmark further demonstrate that FreeLB can be generalized andboost the performance of RoBERTa-large model on other tasks as well. Code isavailable at \url{https://github.com/zhuchen03/FreeLB .