Stacked DeBERT: All Attention in Incomplete Data for Text Classification

  • 2020-01-01 04:49:23
  • Gwenaelle Cunha Sergio, Minho Lee
  • 22

Abstract

In this paper, we propose Stacked DeBERT, short for Stacked DenoisingBidirectional Encoder Representations from Transformers. This novel modelimproves robustness in incomplete data, when compared to existing systems, bydesigning a novel encoding scheme in BERT, a powerful language representationmodel solely based on attention mechanisms. Incomplete data in natural languageprocessing refer to text with missing or incorrect words, and its presence canhinder the performance of current models that were not implemented to withstandsuch noises, but must still perform well even under duress. This is due to thefact that current approaches are built for and trained with clean and completedata, and thus are not able to extract features that can adequately representincomplete data. Our proposed approach consists of obtaining intermediate inputrepresentations by applying an embedding layer to the input tokens followed byvanilla transformers. These intermediate features are given as input to noveldenoising transformers which are responsible for obtaining richer inputrepresentations. The proposed approach takes advantage of stacks of multilayerperceptrons for the reconstruction of missing words' embeddings by extractingmore abstract and meaningful hidden feature vectors, and bidirectionaltransformers for improved embedding representation. We consider two datasetsfor training and evaluation: the Chatbot Natural Language UnderstandingEvaluation Corpus and Kaggle's Twitter Sentiment Corpus. Our model showsimproved F1-scores and better robustness in informal/incorrect texts present intweets and in texts with Speech-to-Text error in the sentiment and intentclassification tasks.

 

Quick Read (beta)

loading the full paper ...