Abstract
Developing Named Entity Recognition (NER) systems for Indian languages hasbeen a long-standing challenge, mainly owing to the requirement of a largeamount of annotated clean training instances. This paper proposes an end-to-endframework for NER for Indian languages in a low-resource setting by exploitingparallel corpora of English and Indian languages and an English NER dataset.The proposed framework includes an annotation projection method that combinesword alignment score and NER tag prediction confidence score on source language(English) data to generate weakly labeled data in a target Indian language. Weemploy a variant of the Teacher-Student model and optimize it jointly on thepseudo labels of the Teacher model and predictions on the generated weaklylabeled data. We also present manually annotated test sets for three Indianlanguages: Hindi, Bengali, and Gujarati. We evaluate the performance of theproposed framework on the test sets of the three Indian languages. Empiricalresults show a minimum 10% performance improvement compared to the zero-shottransfer learning model on all languages. This indicates that weakly labeleddata generated using the proposed annotation projection method in target Indianlanguages can complement well-annotated source language data to enhanceperformance. Our code is publicly available athttps://github.com/aksh555/CL-NERIL