Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability

  • 2020-09-09 04:31:44
  • Mayank Chhipa, Hrushikesh Mahesh Vazurkar, Abhijeet Kumar, Mridul Mishra
  • 1


With the recent influx of bidirectional contextualized transformer languagemodels in the NLP, it becomes a necessity to have a systematic comparativestudy of these models on variety of datasets. Also, the performance of theselanguage models has not been explored on non-GLUE datasets. The study presentedin paper compares the state-of-the-art language models - BERT, ELECTRA and itsderivatives which include RoBERTa, ALBERT and DistilBERT. We conductedexperiments by finetuning these models for cross domain and disparate data andpenned an in-depth analysis of model's performances. Moreover, anexplainability of language models coherent with pretraining is presented whichverifies the context capturing capabilities of these models through a modelagnostic approach. The experimental results establish new state-of-the-art forYelp 2013 rating classification task and Financial Phrasebank sentimentdetection task with 69% accuracy and 88.2% accuracy respectively. Finally, thestudy conferred here can greatly assist industry researchers in choosing thelanguage model effectively in terms of performance or compute efficiency.


Quick Read (beta)

loading the full paper ...