Evaluating the applicability of the transformer-based grammatical error correction system for assessing language accuracy

Abstract

The current study focuses on the use of Grammatical Error Correction (GEC) technology for assessing language accuracy, which has received relatively less attention than complexity and fluency in the context of automated assessment. Adopting a technology-enhanced approach to language assessment, rather than a technology-driven approach, we critically assessed the suitability of the state-of-the-art GEC system for assessing language accuracy in Korean, an understudied language in this regard. We analyzed how reliable this system is quantitatively and what types of error can be generated by this system qualitatively. Also, we used out-of-domain, inclusive data from heritage speakers of Korean, which has never been considered in the development of GEC. Our accuracy analyses show that the system achieves a fairly high accuracy in differentiating between correct and incorrect sentences on our data (F_0.5 = 0.819). However, the system exhibits a tendency to make unnecessary corrections, such as inserting topics/adverbials or correcting particles, while failing to correct ungrammatical ones in some cases. These findings from our mixed-method analyses suggest that language evaluators should recognize the potential for inaccurate assessments when using a GEC system, as its output may be incorrect at this moment, thus highlighting the critical need for digital language assessment literacy.