Abstract
Most of the existing pre-trained language representation models neglect toconsider the linguistic knowledge of texts, which can promote languageunderstanding in NLP tasks. To benefit the downstream tasks in sentimentanalysis, we propose a novel language representation model called SentiLARE,which introduces word-level linguistic knowledge including part-of-speech tagand sentiment polarity (inferred from SentiWordNet) into pre-trained models. Wefirst propose a context-aware sentiment attention mechanism to acquire thesentiment polarity of each word with its part-of-speech tag by queryingSentiWordNet. Then, we devise a new pre-training task called label-aware maskedlanguage model to construct knowledge-aware language representation.Experiments show that SentiLARE obtains new state-of-the-art performance on avariety of sentiment analysis tasks.