Natural Language Inference with Mixed Effects

  • 2020-10-20 17:54:16
  • William Gantt, Benjamin Kane, Aaron Steven White
  • 1

Abstract

There is growing evidence that the prevalence of disagreement in the rawannotations used to construct natural language inference datasets makes thecommon practice of aggregating those annotations to a single label problematic.We propose a generic method that allows one to skip the aggregation step andtrain on the raw annotations directly without subjecting the model to unwantednoise that can arise from annotator response biases. We demonstrate that thismethod, which generalizes the notion of a \textit{mixed effects model} byincorporating \textit{annotator random effects} into any existing neural model,improves performance over models that do not incorporate such effects.

 

Quick Read (beta)

loading the full paper ...