Misspelling Correction with Pre-trained Contextual Language Model

  • 2021-01-08 20:11:01
  • Yifei Hu, Xiaonan Jing, Youlim Ko, Julia Taylor Rayz
  • 0

Abstract

Spelling irregularities, known now as spelling mistakes, have been found forseveral centuries. As humans, we are able to understand most of the misspelledwords based on their location in the sentence, perceived pronunciation, andcontext. Unlike humans, computer systems do not possess the convenient autocomplete functionality of which human brains are capable. While many programsprovide spelling correction functionality, many systems do not take contextinto account. Moreover, Artificial Intelligence systems function in the waythey are trained on. With many current Natural Language Processing (NLP)systems trained on grammatically correct text data, many are vulnerable againstadversarial examples, yet correctly spelled text processing is crucial forlearning. In this paper, we investigate how spelling errors can be corrected incontext, with a pre-trained language model BERT. We present two experiments,based on BERT and the edit distance algorithm, for ranking and selectingcandidate corrections. The results of our experiments demonstrated that whencombined properly, contextual word embeddings of BERT and edit distance arecapable of effectively correcting spelling errors.

 

Quick Read (beta)

loading the full paper ...