Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance

  • 2023-01-25 02:06:13
  • Guijin Son, Hanwool Lee, Nahyeon Kang, Moonjeong Hahm
  • 0


Extraction of sentiment signals from news text, stock message boards, andbusiness reports, for stock movement prediction, has been a rising field ofinterest in finance. Building upon past literature, the most recent worksattempt to better capture sentiment from sentences with complex syntacticstructures by introducing aspect-level sentiment classification (ASC). Despitethe growing interest, however, fine-grained sentiment analysis has not beenfully explored in non-English literature due to the shortage of annotatedfinance-specific data. Accordingly, it is necessary for non-English languagesto leverage datasets and pre-trained language models (PLM) of differentdomains, languages, and tasks to best their performance. To facilitatefinance-specific ASC research in the Korean language, we build KorFinASC, aKorean aspect-level sentiment classification dataset for finance consisting of12,613 human-annotated samples, and explore methods of intermediate transferlearning. Our experiments indicate that past research has been ignorant towardsthe potentially wrong knowledge of financial entities encoded during thetraining phase, which has overestimated the predictive power of PLMs. In ourwork, we use the term "non-stationary knowledge'' to refer to information thatwas previously correct but is likely to change, and present "TGT-Masking'', anovel masking pattern to restrict PLMs from speculating knowledge of the kind.Finally, through a series of transfer learning with TGT-Masking applied weimprove 22.63% of classification accuracy compared to standalone models onKorFinASC.


Quick Read (beta)

loading the full paper ...