ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition

  • 2023-01-21 02:20:43
  • Ling Ge, Chuming Hu, Guanghui Ma, Hong Zhang, Jihong Liu
  • 0

Abstract

For named entity recognition (NER) in zero-resource languages, utilizingknowledge distillation methods to transfer language-independent knowledge fromthe rich-resource source languages to zero-resource languages is an effectivemeans. Typically, these approaches adopt a teacher-student architecture, wherethe teacher network is trained in the source language, and the student networkseeks to learn knowledge from the teacher network and is expected to performwell in the target language. Despite the impressive performance achieved bythese methods, we argue that they have two limitations. Firstly, the teachernetwork fails to effectively learn language-independent knowledge shared acrosslanguages due to the differences in the feature distribution between the sourceand target languages. Secondly, the student network acquires all of itsknowledge from the teacher network and ignores the learning of targetlanguage-specific knowledge. Undesirably, these limitations would hinder themodel's performance in the target language. This paper proposes an unsupervisedprototype knowledge distillation network (ProKD) to address these issues.Specifically, ProKD presents a contrastive learning-based prototype alignmentmethod to achieve class feature alignment by adjusting the distance amongprototypes in the source and target languages, boosting the teacher network'scapacity to acquire language-independent knowledge. In addition, ProKDintroduces a prototypical self-training method to learn the intrinsic structureof the language by retraining the student network on the target data usingsamples' distance information from prototypes, thereby enhancing the studentnetwork's ability to acquire language-specific knowledge. Extensive experimentson three benchmark cross-lingual NER datasets demonstrate the effectiveness ofour approach.

 

Quick Read (beta)

loading the full paper ...