Abstract

Language identification of social media text still remains a challenging taskdue to properties like code-mixing and inconsistent phonetic transliterations.In this paper, we present a supervised learning approach for languageidentification at the word level of low resource Bengali-English code-mixeddata taken from social media. We employ two methods of word encoding, namelycharacter based and root phone based to train our deep LSTM models. Utilizingthese two models we created two ensemble models using stacking and thresholdtechnique which gave 91.78% and 92.35% accuracies respectively on our testingdata.

 

Quick Read (beta)