Abstract

Multilingual search can be achieved with subword tokenization. The accuracyof traditional TF-IDF approaches depend on manually curated tokenization, stopwords and stemming rules, whereas subword TF-IDF (STF-IDF) can offer higheraccuracy without such heuristics. Moreover, multilingual support can beincorporated inherently as part of the subword tokenization model training.XQuAD evaluation demonstrates the advantages of STF-IDF: superior informationretrieval accuracy of 85.4% for English and over 80% for 10 other languageswithout any heuristics-based preprocessing. The software to reproduce theseresults are open-sourced as a part of Text2Text:https://github.com/artitw/text2text

 

Quick Read (beta)