Helpful or Harmful? Comparative Study of Perceived and Actual Effectiveness of LLM-Driven Tutors in Game-Based CFL Learning

Abstract

This study examines the perceived and actual effectiveness of an LLM-driven tutor embedded in an educational game for Chinese as a foreign language (CFL) learners. Drawing on 82 chat sessions from 31 beginner-level (HSK3) CFL learners, we analyzed learners’ satisfaction ratings, accuracy before and after interacting with the tutor, and their post-interaction cognitive behaviors. The results showed that while most sessions received positive or neutral satisfaction scores, actual learning gains were limited, with only marginally significant improvements in accuracy following the learner-tutor interaction. Behavioral analysis further revealed that content-irrelevant responses (e.g., technical guidance) were linked to more effective, higher-level cognitive behaviors, whereas content-relevant responses (e.g., explanations of vocabulary or grammar) were associated with more superficial, less effective behaviors, suggesting a possible over-reliance on the LLM-driven tutor. Regression analyses also confirmed that neither satisfaction nor content relevance significantly predicted long-term behavior patterns. Taken together, these results indicate a disconnect between learners’ positive perceptions of the LLM-driven tutor and their actual learning benefits. This study highlights the need for multi-perspective evaluations of LLM-based educational tools and careful instructional design to avoid unintended cognitive dependence.