Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning

  • 2023-01-30 07:17:09
  • Hyunsoo Cho, Choonghyun Park, Junyeop Kim, Hyuhng Joon Kim, Kang Min Yoo, Sang-goo Lee
  • 0


As the size of the pre-trained language model (PLM) continues to increase,numerous parameter-efficient transfer learning methods have been proposedrecently to compensate for the tremendous cost of fine-tuning. Despite theimpressive results achieved by large pre-trained language models (PLMs) andvarious parameter-efficient transfer learning (PETL) methods on sundrybenchmarks, it remains unclear if they can handle inputs that have beendistributionally shifted effectively. In this study, we systematically explorehow the ability to detect out-of-distribution (OOD) changes as the size of thePLM grows or the transfer methods are altered. Specifically, we evaluatedvarious PETL techniques, including fine-tuning, Adapter, LoRA, andprefix-tuning, on three different intention classification tasks, eachutilizing various language models with different scales.


Quick Read (beta)

loading the full paper ...