Improving Part of Speech Tagging for Learner English

Abstract

As advanced algorithms for natural language processing are being developed, fundamental tasks like part of speech tagging remain crucial and challenging. As a crucial step in natural language processing, part of speech tagging assigns each word in a phrase to the correct part of speech. Spelling mistakes are a frequent blunder we make when utilizing computer software. Spelling mistakes have a negative impact on part of speech tagging performance. The main goal of the study is to use a spell checker to increase the precision of part of speech tagging. In our comparison of four spell checkers—Norvig, Jamspell, Hunspell, and Dummy—Jamspell outperformed the others in terms of accuracy, speed, and the number of words it broke during the correction. This influenced our decision to use Jamspell to correct spelling mistakes. Treebank, Brown, and Conll2000 NLTK corpora were utilized for the project's Part of speech tagging. We applied a softmax function on the hidden state of the LSTM after passing an LSTM over the tokens to make predictions. The outcome is a vector of tag scores from which we can determine the projected tag for a word based on the highest value in this distribution of tag scores. The result shows an accuracy of 97%, a macro average of 98% and a weighted average of 97% when tested on 212 tagged sentences after spelling correction was applied.