The Robustness of AI-Classifiers in the Face of AI-Assisted Plagiarism: The Case of Turnitin AI Content Detector

Abstract

The release of ChatGPT marked the beginning of a new era of artificial intelligence (AI)-assisted plagiarism that disrupted traditional assessment practices. In response to this existential threat, Turnitin added an AI content detector to its platform, which is the most used plagiarism detection software in most academic institutions. However, several early studies suggest that AI detectors have questionable accuracy rates, especially when dealing with English as a second language writings or modified AI texts. To explore this uncharted territory in second language writing research, the present study evaluated the performance of Turnitin AI Detector with different text types. Using a comparative descriptive research paradigm, the researchers measured and compared Turnitin's detection accuracy in four conditions: a) English as a foreign language texts, (b) AI-generated texts, (c) paraphrased AI-generated texts, and (d) humanized AI-generated texts. The results demonstrated that Turnitin had an above-average accuracy in detecting AI texts, but its accuracy dropped slightly for paraphrased text and significantly for humanized texts.