Temporal-aware Language Representation Learning From Crowdsourced Labels

  • 2021-07-15 05:25:56
  • Yang Hao, Xiao Zhai, Wenbiao Ding, Zitao Liu
  • 0

Abstract

Learning effective language representations from crowdsourced labels iscrucial for many real-world machine learning tasks. A challenging aspect ofthis problem is that the quality of crowdsourced labels suffer high intra- andinter-observer variability. Since the high-capacity deep neural networks caneasily memorize all disagreements among crowdsourced labels, directly applyingexisting supervised language representation learning algorithms may yieldsuboptimal solutions. In this paper, we propose \emph{TACMA}, a\underline{t}emporal-\underline{a}ware language representation learningheuristic for \underline{c}rowdsourced labels with \underline{m}ultiple\underline{a}nnotators. The proposed approach (1) explicitly models theintra-observer variability with attention mechanism; (2) computes andaggregates per-sample confidence scores from multiple workers to address theinter-observer disagreements. The proposed heuristic is extremely easy toimplement in around 5 lines of code. The proposed heuristic is evaluated onfour synthetic and four real-world data sets. The results show that ourapproach outperforms a wide range of state-of-the-art baselines in terms ofprediction accuracy and AUC. To encourage the reproducible results, we make ourcode publicly available at \url{https://github.com/CrowdsourcingMining/TACMA}.

 

Quick Read (beta)

loading the full paper ...